Six steps, and you can audit every one.
Secondary analysis turns raw sequencing reads into an interpretable answer. Every stage below is a real process with a pinned tool version, so the run that produced a result can be reproduced exactly — which is a regulatory requirement, not a convenience.
01 · The pipeline
From FASTQ to called variants.
Two of the six stages run on FPGA hardware. Those two dominate the wall clock, which is why accelerating them moves the end-to-end number.
- 01
Upload
Presigned multipart transfer straight to object storage. Resumable, checksum-verified.
- 02
Validate
FASTQ format, paired-end pairing, read length and quality encoding, before compute is spent.
- 03
Align
BWA-MEM alignment against GRCh38.
FPGA - 04
Call variants
Variant calling to VCF.
FPGA - 05
Quality control
Coverage, duplication and threshold evaluation. Below-threshold samples are flagged.
- 06
Deliver
Variants, metrics and artifacts — queryable, paginated, exportable.
02 · Reproducibility
Why every run is pinned.
The pipeline is a containerised Nextflow DSL2 workflow whose images are pinned by content digest. A pipeline version is therefore byte-identical on every run, and the reference set it used is recorded explicitly rather than resolved as “latest”.
Analyses are append-only once terminal. A re-run creates a new analysis record pointing at the one it supersedes, so history is never rewritten. When a regulator asks which pipeline version produced a given result, the answer is a digest in an immutable record.
Every process checkpoints to object storage, so a reclaimed spot instance resumes from the last completed step rather than restarting a multi-hour run.
03 · Architecture
Two planes, joined by a queue.
The control plane is everything a customer touches: authentication, organisations and members, projects and samples, uploads, orchestration, billing, reporting and the web surface in twenty languages.
The execution plane is where the bioinformatics lives — Nextflow and Python on F2 hardware, with GATK, BWA and samtools. Bioinformatics is a Python monoculture and fighting that is a losing battle.
The boundary is a versioned job specification written to a queue. Neither side imports the other. This is not architectural fashion: it means the web application and the scientific pipeline can be developed, deployed, scaled and — critically — validated independently.