bench-perf-oss
Compare BioIR vs OSS folding performance: model.forward() latency, GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.name: bench-perf-ossdescription: >- Compare BioIR vs OSS folding performance: model.forward() latency, GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.license: Apache-2.0metadata: author: NVIDIA CorporationBenchmark BioIR vs OSS e2e model.forward
Input: model key + OSS source path. Output: WORKDIR harnesses, locked
config, per-sample model.forward() latency + GPU artifacts, OpenStructure
lDDT and DockQ vs benchmarks/dataset/ground_truth/,
comparison report.
This file defines scope, mandatory gates, and phase handoffs. Follow workflow.md for procedures; open references when their phase starts.
This is v1 for structure folding only (keys with a coordinate
head: Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3, protenix-v2).
The headline compared numbers are GPU-synced model.forward()
time and GPU memory. Parse, tokenize, featurize, H2D, postprocess,
write, and scoring are outside the timing window.
Both quality metrics are required: OpenStructure lDDT on every sample, and DockQ on supported protein-protein or protein-small-molecule complexes. Every row records DockQ applicability; DockQ 2.1.3 does not support RNA/DNA interfaces. Whole-structure lDDT and interface DockQ answer different questions. See measurement.md.
Do not run this playbook unchanged on a non-folding key
(boltz-2-affinity, or anything whose output is not a structure).
See Other model kinds.
Templates on both sides or neither. Retain template-bearing spec items and verify attachment on both featurized batches before timing; missing templates change both quality and workload. Follow the model profile and samples.md.
Serial, one sample per call. executor_backend=None. Feed
processor([one_record]) — never the whole manifest as one batch
(host-RAM OOM). Same one-at-a-time rule on the OSS side.
Choose a path from the support matrix
(docs/ref/support-matrix.md, Pipeline column):
- Processor path —
build_processor. The model has a factory (Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3,protenix-v2). BioIR uses serialbuild_processor; OSS uses its own e2e. Each side featurizes itself. - OSS feature path — OSS pipeline + BioIR module. A folding model with a
compute path but no pipeline factory. No current key uses it. Use the
OSS data pipeline for features, then replace the OSS
nn.Modulewith the BioIR module. See no-pipeline.md.boltz-2-affinityis not a folding bench — see Other model kinds.
Authorities:
- Phase procedures — workflow.md
- Production API —
docs/ref/api.md(build_processor,profile_inference,runtime_args) - Serial vs Ray —
docs/ref/architecture.md - Model keys and MSA rules —
docs/ref/support-matrix.md - Checkpoints —
docs/ref/model-weights.md - Dataset and sample catalog — samples.md (built by
rebuild_dataset.py) - Environment — environment.md
- Timing protocol — measurement.md
- MSA mapping (dataset -> OSS) — msa.md
- Template mapping (dataset -> OSS) — templates.md
- No BioIR pipeline — no-pipeline.md
- DeepSpeed evoformer (OF2 / OF3) — deepspeed-evoformer.md
- Model profiles — Boltz-2, OpenFold2 / AlphaFold2, OpenFold3, Protenix-v2
- Portable SKU package (optional) — bundle.md
Mandatory gates
If any gate cannot be satisfied, stop and report the blocker. Do not claim a complete bench.
- Both environments are probed and isolated first. Check BioIR and OSS imports, set up whatever is missing, and resolve package conflicts before any timed run. See environment.md. Do not install OSS pins into the BioIR interpreter.
- OSS revision is the pin. Use the exact revision selected by the
model profile; when the profile selects a checked-in submodule
gitlink, that gitlink is the pin. Boltz-1/2 =
v2.2.1. OpenFold2 = aqlaboratory/openfoldv2.2.0. Amaincheckout or a different tag is a hard failure. See environment.md. A recorded, reverted, input-side patch on top of the pin is allowed — only to make both sides honour the same inputs, never to change the compute path (templates.md). - No invented numbers. Every latency, GPU power, clock-rate, or speedup figure in the report cites an executed command, exit code, and artifact path. Parse tool output — never estimate.
- Dataset is the one
rebuild_dataset.pybuilds, only. The builder ships in this skill atdataset/; local build root:benchmarks/dataset/. Usespec_full.jsonorspec_monomer.json(samples.md). Do not substituteexamples/data/samples/, FASTA-only inputs, or a synthetic residue sweep unless the user explicitly replaces the dataset. - MSAs and templates are mandatory when they exist. For every
polymer that lists an unpaired or paired A3M (or Boltz CSV) or a
template in the spec, both BioIR and OSS must load those files.
Running without MSAs, with
use_msa_server, or with empty protein MSAs when files exist is a hard failure. - Templates are custom templates: all of them, both sides. The spec's templates are caller-supplied, so both sides featurize every one, limited only by a cap set identically on each side. Turn off the similarity, coverage, date, and keep/drop gates that exist to filter search output; patch a side if config cannot. Verify by counting populated template slots per side — a non-zero mask passes while three of four templates are missing. See MSA and template contract, msa.md, and templates.md.
- Same sample set on both sides. Write
$WORKDIR/ref_data/sample_manifest.jsonbefore any timed run. Both harnesses assert exact set equality against it. - The OSS input mapping is a script plus an index, not ad-hoc
files. When the OSS loader needs another layout or naming, build
$WORKDIR/oss_data/withbench/stage_oss_data.pyand recordoss_data/index.json; the harness reads the index. Symlink the dataset, never copy or edit it, and assert every declared input staged with one distinct key per chain (Phase 2). - Config parity is locked first. Write
$WORKDIR/ref_data/bench_config.jsonbefore Phase 4. Recycling, sampling, diffusion samples, precision, seed, and GPU id must match. - BioIR starts from its optimized pretrained config. Omit
config=and processorengine_kwargs["config"]unless applying documented model-profile overrides for input parity, fixed recycle counts, or approximate options. Derive those configs withget_pretrained_config, change only the documented fields, and record the delta. An eager arm additionally callsdisable_cuda_graphs(). Never hand-build aBaseConfig, copy OSS layer configs onto BioIR, or replace graph policies. See measurement.md. - Serial, one GPU, one sample per call.
executor_backend=None. PinCUDA_VISIBLE_DEVICESto one device. Loop the in-scope manifest and callprocessor([one_record])(or OSS featurize +forwardon that one batch). Neverprocessor(all_rows)— feature generation then holds every record in host RAM and OOM-kills before the engine. See measurement.md. - Every timed column runs on the same host and the same GPU. Run
BioIR and every OSS column on one physical machine and one device,
not just the same GPU model. Record the hostname, GPU UUID, and
driver version in
ref_data/bench_config.json. Latency from different hosts is not comparable, because CPU, driver, power limit, and thermals differ. If a column must be rerun elsewhere, rerun every column there. - Both scorers are installed before timing. Give
ostits own conda env andDockQits own venv, separate from BioIR, OSS and Miniforge base. Score written CIFs withost compare-structuresand supported interfaces withDockQ. Every row getsdockq_status: RNA/DNA interfaces useunsupported_chain_types; a monomer usesnull/single_chain, never0.0. See environment.md and environment.md. - OSS uses its recommended inference kernels, and the config has
to say so. Do not strip an optimized backend "for fairness", and
do not assume a shipped default is the recommendation; resolve the
model profile and upstream kernel documentation. If the selected
profile requires cuEquivariance, enable it in config—installing the
wheel is not enabling the kernel. If it requires DeepSpeed
Evoformer attention, build only that op from source
(
DS_BUILD_OPS=0 DS_BUILD_EVOFORMER_ATTN=1), non-editable and never-e, budgeting 5–10 min for the CUTLASS compile. See environment.md and deepspeed-evoformer.md. A fallback is a different column, and only with user approval. - Warmup 1, measure 1. One discarded forward (JIT / graph capture
/
torch.compile), then one timed forward. That single time is the headline. Do not average or take a median. - Five samples before the full sweep. After preparation, run
BioIR and OSS eager on five samples spanning the residue range,
run the fast synthetic direct-module compile probe, then run OSS
torch.compileon the same five when the compile retry ladder succeeds. Show the measured results and ask whether to proceed. Do not launch the full sweep without the user's explicit approval (five-sample smoke gate). - No hidden skips. A missing sample, missing MSA, OOM, or crash is
recorded. Do not
continuepast it silently. - Implementation notes are mandatory. Keep
$WORKDIR/implementation-notes.mdcurrent as work happens.
Approximate options
Some BioIR options are approximate: they change the sampled coordinates
instead of reproducing the exact path bit for bit. Frame-aware prediction
reuse, frame_reuse_period, is one; Protenix-v2, OpenFold3 P2 and Boltz-2
have it. Its pretrained default is "auto" for Protenix-v2 and OpenFold3 P2,
which reuses at the step counts they were evaluated with, and 0 (exact) for
Boltz-2. Rules:
- Default on for Protenix-v2 and OpenFold3 P2. Both profiles lock 200
diffusion steps, a count the option was evaluated with, so their BioIR arm
runs with
frame_reuse_period = 4. Set 4 explicitly rather than relying on"auto": a forced 4 raises where"auto"would quietly run exact. Build the config from the model's pretrained config (the profile shows how), change only that field, pass it as the engine config (engine_kwargs["config"]), and record the value inbench_config.json. Boltz-2 keeps the option off; add its approximate column only when the user asks (its profile shows how). - The exact arm sets
frame_reuse_period = 0. Build it the same way. The pretrained Protenix-v2 and OpenFold3 P2 configs reuse, so leaving them unchanged does not give an exact column. The harness written in Phase 3 takes an--exactswitch that sets 0. - Run the exact arm as well. Add one BioIR column with the option off (same seed, same inputs, same GPU), time it like every other column, and report the approximate column's speedup against it and against OSS.
- Judge fidelity on paired scores, not equality. Score both BioIR columns with the same lDDT and DockQ scorers and report the paired per-sample difference (approximate minus exact) next to each speedup. Bit equality is required only of the exact column and of the option when off.
- Label the column. Name the column
bioir_approx_<option>and list the option and value in the table caption. Never present the approximate column as the exact BioIR result.
MSA and template contract
Mapping the dataset's alignments and templates onto an OSS tree is its own job, with its own silent failure modes. Full procedure, per-tree mappings, and verification: msa.md and templates.md. The gates below are the short form.
Protein unpaired MSA is required for Boltz-1/2 and OpenFold3, and
for every AlphaFold2 / OpenFold2 key (docs/ref/support-matrix.md)
when the spec lists A3Ms. Paired MSA is used when the spec lists
paired_msas. RNA, DNA, and ligand chains carry empty msas —
that is correct; do not invent files.
BioIR load (processor path). Convert each in-scope spec item to an
InputRequest with paths resolved against
benchmarks/dataset/
(samples.md). Do not call
load_requests on spec_*.json. Attach every listed A3M and
template.
OSS load (both paths). Map the same spec item to the OSS
inference script. Boltz: boltz_yaml + boltz_msa_csv, rewriting
stale absolute msa: paths to $DATASET_ROOT/casp15/msa/. Other
OSS trees: spec polymers + resolved A3Ms. Do not call
build_processor on the OSS feature path.
After load, before any forward, assert all of:
- Every spec
msas/paired_msas(and Boltz CSV) path exists on disk under$DATASET_ROOT. - Every protein polymer whose spec lists MSAs has a non-empty
MSARecordlist on the BioIRInputRequest(processor path) and non-empty alignments on the OSS featurizer input. - Every A3M the spec lists for that item is attached (unpaired and paired). A listed file that is skipped is a hard failure.
- The OSS adapter consumes those same alignments. No ColabFold / MSA server. No FASTA-only rewrite that drops alignments.
- Every template the spec lists is attached on both sides:
populated template slots equal
min(n_supplied, cap)on each, with the same entry and chain per slot (templates.md).
Phases
Run the phases in order. Each names what to read, what it must leave in
$WORKDIR, and the check that ends it; do not start a phase before the
previous exit check passes. Harnesses live in $WORKDIR; source changes are
limited to the recorded input-side patches allowed by the mandatory gates.
Phase 0 — WORKDIR, environments, GPU, OSS root
- Read:
workflow.md,
environment.md, and the model profile under
models/. OpenFold2 / OpenFold3 also need deepspeed-evoformer.md. - Produce: the
$WORKDIRtree,$OSS_ROOTat the pin, the BioIR and OSS interpreters,envs/ostandenvs/dockq,ref_data/gpu_inventory.csv, and the environment fields plusfeature_path(processor|oss) inref_data/bench_config.json. - Exit: both sides import on the same GPU, and the checkpoint strict-loads once in each interpreter from a recorded local path with the hub untouched (weights preflight). A non-folding key stops here.
Phase 1 — Manifest + locked config
- Read: workflow.md, samples.md, measurement.md, and the profile's runtime lock.
- Produce:
ref_data/sample_manifest.jsonwith every spec item, the in-scope filter by model capability only, and every A3M and template resolved;ref_data/bench_config.jsonwith the locked runtime args, seed, precision, GPU, dataset digests, BioIR default constructor and graph activation, OSS entry script, and compile plan. - Exit: the OSS script's expanded effective config matches the lock. The config is read-only from here.
Phase 2 — Map the dataset into oss_data
- Read: workflow.md, msa.md, templates.md, and the profile's mapping section.
- Produce:
bench/stage_oss_data.py, anoss_data/tree of symlinks, andoss_data/index.jsonper sample and chain. - Exit: a rerun is idempotent and silent, every declared input is staged with one distinct key per chain, and one monomer and one multi-chain sample are checked by hand.
Phase 3 — Write the harnesses
- Read: workflow.md and
measurement.md. Copy the shared helpers into
bench/: timing.py, gpu_telemetry.py, scoring.py, check_results.py, summarize_speedup.py, and plot_latency.py. Do not rewrite them. The OSS feature path also needs no-pipeline.md. - Produce: on the processor path,
bench/run_bioir.pyandbench/run_oss.py; on the OSS feature path,bench/dump_oss_features.pyplus both runners. Every loop shows a progress bar outside the timing window. - Exit: the five-sample smoke gate passes, its results are shown, and the user explicitly approves the full sweep.
Phase 4 — Run BioIR
- Read: workflow.md.
- Produce:
results/bioir_forward.jsonand logs underdebug/. The OSS feature path writes the feature dumps first. Run with$BIOIR_PYTHONandCUTEDSL_FORCE_CUBIN=1.
Phase 5 — Run OSS e2e
- Read: workflow.md.
- Produce:
results/oss_eager.jsonalways, andresults/oss_compile.jsononly after a passing probe or retry. Same manifest, locked knobs, and GPU; the OSS feature path loads the Phase 4 dumps. Run with$OSS_PYTHON. An OSS eager kernel crash blocks that backend; never relabel a fallback asoss_e2e.
Phase 6 — Compare and report
- Read: workflow.md and measurement.md.
- Produce:
results/speedup.jsonand both charts, then print setup, dataset, the latency table, speedup geomean and median overall and per residue bin, lDDT, DockQ, GPU power and clocks from in-forward samples, commands, and the path toimplementation-notes.md. Show the chart PNGs inline. - Exit:
harness/check_results.pypasses on every result file, every number cites an artifact, and latency is in seconds to three decimals.
Optional — compact a portable bioir-perf package
Skip unless the user asks to pack completed benches so they can run on another GPU SKU or in a fresh container. Follow bundle.md.
The pack is bioir-perf/, never release/, with one directory per model.
Completion requires a GPU smoke of every included model's BioIR, OSS eager,
and supported compile arms (Protenix never compiles), plus speedup.json,
comparison_report.md, and per-column JSON. This smoke checks harnesses;
a full-dataset SKU sweep is separate. Supported keys: boltz2, of3,
protenix, of2.
Boltz-2 — MSA deletion bug
The pinned OSS v2.2.1 featurizer zeroes non-query MSA deletions, and BioIR
does not. A large BioIR vs OSS lDDT gap on MSA-bearing proteins may be that,
not the engine. Never zero BioIR deletions to match; triage it with
models/boltz2.md.
Other model kinds
v1 is a folding bench. If the user names a non-folding key
(boltz-2-affinity, embeddings, design), stop and extend this skill (or a
sibling) before writing harnesses. What carries over and what each new kind
must redefine is in
workflow.md.
Out of scope (v1)
- Nsight Systems / Nsight Compute, MFU, SOL%, kernel breakdown
- Ray replica throughput (large-dataset path; not this spec)
- Template search (HHsearch / HMMsearch); templates are attached from the spec's shipped structures only (see samples.md)
- The demo tree
examples/data/samples/as a substitute dataset - DockQ / feature-tensor equivalence (
make-data-pipeline) - Isolated same-features microbench of a single stack (Benchmark the Swap
in
docs/advanced/accelerate-custom-model.md) — the OSS feature path is e2emodel.forwardon OSS features, not a layer bench - Non-folding heads (
boltz-2-affinity, embeddings, design, …) until this skill (or a sibling) is extended
Those are separate skills or a later revision.
Key gotchas
- Launch each harness with its own python —
$BIOIR_PYTHONor$OSS_PYTHON, never a barepythonafter Phase 0. CUTEDSL_FORCE_CUBIN=1on every BioIR process. JIT compile time makes forward latency unstable.- No
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, ever — it can slowforward(measurement.md). - CUDA 12 packages → CUDA 13. Install the
cu13counterpart of anycu12pin (environment.md). - Protenix-v2 is Processor path. Never pass
return_full_data=False;build_processorrejects it. Compare quality over seeds or the sample set because OSS adds Monte Carlo pair dropout that BioIR omits. - Seed goes on the feature-generator stage, not the tokenizer
(
docs/ref/api.md). - Paired A3Ms are per chain with equal row counts (
docs/ref/api.md). - Let DockQ choose the chain mapping and record it.
- Audit graph execution on every forward. Follow the counters and
cache rules in measurement.md; the default
model captures graphs without an
optimize()call. - Sharing the GPU with another job invalidates the run.