← LeaderboardThe Skill Leaderboard

bench-perf-oss

Compare BioIR vs OSS folding performance: model.forward() latency, GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.

Published by NVIDIA ↗NVIDIA/skills

Try your first task

A source-grounded starter task has not been verified for this skill yet.

SKILL.md

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.# SPDX-License-Identifier: Apache-2.0## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.name: bench-perf-ossdescription: >-  Compare BioIR vs OSS folding performance: model.forward() latency,  GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup,  execution or audits; excludes affinity heads and Ray throughput.license: Apache-2.0metadata:   author: NVIDIA Corporation

Benchmark BioIR vs OSS e2e model.forward

Input: model key + OSS source path. Output: WORKDIR harnesses, locked config, per-sample model.forward() latency + GPU artifacts, OpenStructure lDDT and DockQ vs benchmarks/dataset/ground_truth/, comparison report.

This file defines scope, mandatory gates, and phase handoffs. Follow workflow.md for procedures; open references when their phase starts.

This is v1 for structure folding only (keys with a coordinate head: Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3, protenix-v2). The headline compared numbers are GPU-synced model.forward() time and GPU memory. Parse, tokenize, featurize, H2D, postprocess, write, and scoring are outside the timing window.

Both quality metrics are required: OpenStructure lDDT on every sample, and DockQ on supported protein-protein or protein-small-molecule complexes. Every row records DockQ applicability; DockQ 2.1.3 does not support RNA/DNA interfaces. Whole-structure lDDT and interface DockQ answer different questions. See measurement.md.

Do not run this playbook unchanged on a non-folding key (boltz-2-affinity, or anything whose output is not a structure). See Other model kinds.

Templates on both sides or neither. Retain template-bearing spec items and verify attachment on both featurized batches before timing; missing templates change both quality and workload. Follow the model profile and samples.md.

Serial, one sample per call. executor_backend=None. Feed processor([one_record]) — never the whole manifest as one batch (host-RAM OOM). Same one-at-a-time rule on the OSS side.

Choose a path from the support matrix (docs/ref/support-matrix.md, Pipeline column):

  • Processor path — build_processor. The model has a factory (Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3, protenix-v2). BioIR uses serial build_processor; OSS uses its own e2e. Each side featurizes itself.
  • OSS feature path — OSS pipeline + BioIR module. A folding model with a compute path but no pipeline factory. No current key uses it. Use the OSS data pipeline for features, then replace the OSS nn.Module with the BioIR module. See no-pipeline.md. boltz-2-affinity is not a folding bench — see Other model kinds.

Authorities:

Mandatory gates

If any gate cannot be satisfied, stop and report the blocker. Do not claim a complete bench.

  1. Both environments are probed and isolated first. Check BioIR and OSS imports, set up whatever is missing, and resolve package conflicts before any timed run. See environment.md. Do not install OSS pins into the BioIR interpreter.
  2. OSS revision is the pin. Use the exact revision selected by the model profile; when the profile selects a checked-in submodule gitlink, that gitlink is the pin. Boltz-1/2 = v2.2.1. OpenFold2 = aqlaboratory/openfold v2.2.0. A main checkout or a different tag is a hard failure. See environment.md. A recorded, reverted, input-side patch on top of the pin is allowed — only to make both sides honour the same inputs, never to change the compute path (templates.md).
  3. No invented numbers. Every latency, GPU power, clock-rate, or speedup figure in the report cites an executed command, exit code, and artifact path. Parse tool output — never estimate.
  4. Dataset is the one rebuild_dataset.py builds, only. The builder ships in this skill at dataset/; local build root: benchmarks/dataset/. Use spec_full.json or spec_monomer.json (samples.md). Do not substitute examples/data/samples/, FASTA-only inputs, or a synthetic residue sweep unless the user explicitly replaces the dataset.
  5. MSAs and templates are mandatory when they exist. For every polymer that lists an unpaired or paired A3M (or Boltz CSV) or a template in the spec, both BioIR and OSS must load those files. Running without MSAs, with use_msa_server, or with empty protein MSAs when files exist is a hard failure.
  6. Templates are custom templates: all of them, both sides. The spec's templates are caller-supplied, so both sides featurize every one, limited only by a cap set identically on each side. Turn off the similarity, coverage, date, and keep/drop gates that exist to filter search output; patch a side if config cannot. Verify by counting populated template slots per side — a non-zero mask passes while three of four templates are missing. See MSA and template contract, msa.md, and templates.md.
  7. Same sample set on both sides. Write $WORKDIR/ref_data/sample_manifest.json before any timed run. Both harnesses assert exact set equality against it.
  8. The OSS input mapping is a script plus an index, not ad-hoc files. When the OSS loader needs another layout or naming, build $WORKDIR/oss_data/ with bench/stage_oss_data.py and record oss_data/index.json; the harness reads the index. Symlink the dataset, never copy or edit it, and assert every declared input staged with one distinct key per chain (Phase 2).
  9. Config parity is locked first. Write $WORKDIR/ref_data/bench_config.json before Phase 4. Recycling, sampling, diffusion samples, precision, seed, and GPU id must match.
  10. BioIR starts from its optimized pretrained config. Omit config= and processor engine_kwargs["config"] unless applying documented model-profile overrides for input parity, fixed recycle counts, or approximate options. Derive those configs with get_pretrained_config, change only the documented fields, and record the delta. An eager arm additionally calls disable_cuda_graphs(). Never hand-build a BaseConfig, copy OSS layer configs onto BioIR, or replace graph policies. See measurement.md.
  11. Serial, one GPU, one sample per call. executor_backend=None. Pin CUDA_VISIBLE_DEVICES to one device. Loop the in-scope manifest and call processor([one_record]) (or OSS featurize + forward on that one batch). Never processor(all_rows) — feature generation then holds every record in host RAM and OOM-kills before the engine. See measurement.md.
  12. Every timed column runs on the same host and the same GPU. Run BioIR and every OSS column on one physical machine and one device, not just the same GPU model. Record the hostname, GPU UUID, and driver version in ref_data/bench_config.json. Latency from different hosts is not comparable, because CPU, driver, power limit, and thermals differ. If a column must be rerun elsewhere, rerun every column there.
  13. Both scorers are installed before timing. Give ost its own conda env and DockQ its own venv, separate from BioIR, OSS and Miniforge base. Score written CIFs with ost compare-structures and supported interfaces with DockQ. Every row gets dockq_status: RNA/DNA interfaces use unsupported_chain_types; a monomer uses null / single_chain, never 0.0. See environment.md and environment.md.
  14. OSS uses its recommended inference kernels, and the config has to say so. Do not strip an optimized backend "for fairness", and do not assume a shipped default is the recommendation; resolve the model profile and upstream kernel documentation. If the selected profile requires cuEquivariance, enable it in config—installing the wheel is not enabling the kernel. If it requires DeepSpeed Evoformer attention, build only that op from source (DS_BUILD_OPS=0 DS_BUILD_EVOFORMER_ATTN=1), non-editable and never -e, budgeting 5–10 min for the CUTLASS compile. See environment.md and deepspeed-evoformer.md. A fallback is a different column, and only with user approval.
  15. Warmup 1, measure 1. One discarded forward (JIT / graph capture / torch.compile), then one timed forward. That single time is the headline. Do not average or take a median.
  16. Five samples before the full sweep. After preparation, run BioIR and OSS eager on five samples spanning the residue range, run the fast synthetic direct-module compile probe, then run OSS torch.compile on the same five when the compile retry ladder succeeds. Show the measured results and ask whether to proceed. Do not launch the full sweep without the user's explicit approval (five-sample smoke gate).
  17. No hidden skips. A missing sample, missing MSA, OOM, or crash is recorded. Do not continue past it silently.
  18. Implementation notes are mandatory. Keep $WORKDIR/implementation-notes.md current as work happens.

Approximate options

Some BioIR options are approximate: they change the sampled coordinates instead of reproducing the exact path bit for bit. Frame-aware prediction reuse, frame_reuse_period, is one; Protenix-v2, OpenFold3 P2 and Boltz-2 have it. Its pretrained default is "auto" for Protenix-v2 and OpenFold3 P2, which reuses at the step counts they were evaluated with, and 0 (exact) for Boltz-2. Rules:

  1. Default on for Protenix-v2 and OpenFold3 P2. Both profiles lock 200 diffusion steps, a count the option was evaluated with, so their BioIR arm runs with frame_reuse_period = 4. Set 4 explicitly rather than relying on "auto": a forced 4 raises where "auto" would quietly run exact. Build the config from the model's pretrained config (the profile shows how), change only that field, pass it as the engine config (engine_kwargs["config"]), and record the value in bench_config.json. Boltz-2 keeps the option off; add its approximate column only when the user asks (its profile shows how).
  2. The exact arm sets frame_reuse_period = 0. Build it the same way. The pretrained Protenix-v2 and OpenFold3 P2 configs reuse, so leaving them unchanged does not give an exact column. The harness written in Phase 3 takes an --exact switch that sets 0.
  3. Run the exact arm as well. Add one BioIR column with the option off (same seed, same inputs, same GPU), time it like every other column, and report the approximate column's speedup against it and against OSS.
  4. Judge fidelity on paired scores, not equality. Score both BioIR columns with the same lDDT and DockQ scorers and report the paired per-sample difference (approximate minus exact) next to each speedup. Bit equality is required only of the exact column and of the option when off.
  5. Label the column. Name the column bioir_approx_<option> and list the option and value in the table caption. Never present the approximate column as the exact BioIR result.

MSA and template contract

Mapping the dataset's alignments and templates onto an OSS tree is its own job, with its own silent failure modes. Full procedure, per-tree mappings, and verification: msa.md and templates.md. The gates below are the short form.

Protein unpaired MSA is required for Boltz-1/2 and OpenFold3, and for every AlphaFold2 / OpenFold2 key (docs/ref/support-matrix.md) when the spec lists A3Ms. Paired MSA is used when the spec lists paired_msas. RNA, DNA, and ligand chains carry empty msas — that is correct; do not invent files.

BioIR load (processor path). Convert each in-scope spec item to an InputRequest with paths resolved against benchmarks/dataset/ (samples.md). Do not call load_requests on spec_*.json. Attach every listed A3M and template.

OSS load (both paths). Map the same spec item to the OSS inference script. Boltz: boltz_yaml + boltz_msa_csv, rewriting stale absolute msa: paths to $DATASET_ROOT/casp15/msa/. Other OSS trees: spec polymers + resolved A3Ms. Do not call build_processor on the OSS feature path.

After load, before any forward, assert all of:

  1. Every spec msas / paired_msas (and Boltz CSV) path exists on disk under $DATASET_ROOT.
  2. Every protein polymer whose spec lists MSAs has a non-empty MSARecord list on the BioIR InputRequest (processor path) and non-empty alignments on the OSS featurizer input.
  3. Every A3M the spec lists for that item is attached (unpaired and paired). A listed file that is skipped is a hard failure.
  4. The OSS adapter consumes those same alignments. No ColabFold / MSA server. No FASTA-only rewrite that drops alignments.
  5. Every template the spec lists is attached on both sides: populated template slots equal min(n_supplied, cap) on each, with the same entry and chain per slot (templates.md).

Phases

Run the phases in order. Each names what to read, what it must leave in $WORKDIR, and the check that ends it; do not start a phase before the previous exit check passes. Harnesses live in $WORKDIR; source changes are limited to the recorded input-side patches allowed by the mandatory gates.

Phase 0 — WORKDIR, environments, GPU, OSS root

  • Read: workflow.md, environment.md, and the model profile under models/. OpenFold2 / OpenFold3 also need deepspeed-evoformer.md.
  • Produce: the $WORKDIR tree, $OSS_ROOT at the pin, the BioIR and OSS interpreters, envs/ost and envs/dockq, ref_data/gpu_inventory.csv, and the environment fields plus feature_path (processor | oss) in ref_data/bench_config.json.
  • Exit: both sides import on the same GPU, and the checkpoint strict-loads once in each interpreter from a recorded local path with the hub untouched (weights preflight). A non-folding key stops here.

Phase 1 — Manifest + locked config

  • Read: workflow.md, samples.md, measurement.md, and the profile's runtime lock.
  • Produce: ref_data/sample_manifest.json with every spec item, the in-scope filter by model capability only, and every A3M and template resolved; ref_data/bench_config.json with the locked runtime args, seed, precision, GPU, dataset digests, BioIR default constructor and graph activation, OSS entry script, and compile plan.
  • Exit: the OSS script's expanded effective config matches the lock. The config is read-only from here.

Phase 2 — Map the dataset into oss_data

  • Read: workflow.md, msa.md, templates.md, and the profile's mapping section.
  • Produce: bench/stage_oss_data.py, an oss_data/ tree of symlinks, and oss_data/index.json per sample and chain.
  • Exit: a rerun is idempotent and silent, every declared input is staged with one distinct key per chain, and one monomer and one multi-chain sample are checked by hand.

Phase 3 — Write the harnesses

Phase 4 — Run BioIR

  • Read: workflow.md.
  • Produce: results/bioir_forward.json and logs under debug/. The OSS feature path writes the feature dumps first. Run with $BIOIR_PYTHON and CUTEDSL_FORCE_CUBIN=1.

Phase 5 — Run OSS e2e

  • Read: workflow.md.
  • Produce: results/oss_eager.json always, and results/oss_compile.json only after a passing probe or retry. Same manifest, locked knobs, and GPU; the OSS feature path loads the Phase 4 dumps. Run with $OSS_PYTHON. An OSS eager kernel crash blocks that backend; never relabel a fallback as oss_e2e.

Phase 6 — Compare and report

  • Read: workflow.md and measurement.md.
  • Produce: results/speedup.json and both charts, then print setup, dataset, the latency table, speedup geomean and median overall and per residue bin, lDDT, DockQ, GPU power and clocks from in-forward samples, commands, and the path to implementation-notes.md. Show the chart PNGs inline.
  • Exit: harness/check_results.py passes on every result file, every number cites an artifact, and latency is in seconds to three decimals.

Optional — compact a portable bioir-perf package

Skip unless the user asks to pack completed benches so they can run on another GPU SKU or in a fresh container. Follow bundle.md.

The pack is bioir-perf/, never release/, with one directory per model. Completion requires a GPU smoke of every included model's BioIR, OSS eager, and supported compile arms (Protenix never compiles), plus speedup.json, comparison_report.md, and per-column JSON. This smoke checks harnesses; a full-dataset SKU sweep is separate. Supported keys: boltz2, of3, protenix, of2.

Boltz-2 — MSA deletion bug

The pinned OSS v2.2.1 featurizer zeroes non-query MSA deletions, and BioIR does not. A large BioIR vs OSS lDDT gap on MSA-bearing proteins may be that, not the engine. Never zero BioIR deletions to match; triage it with models/boltz2.md.

Other model kinds

v1 is a folding bench. If the user names a non-folding key (boltz-2-affinity, embeddings, design), stop and extend this skill (or a sibling) before writing harnesses. What carries over and what each new kind must redefine is in workflow.md.

Out of scope (v1)

  • Nsight Systems / Nsight Compute, MFU, SOL%, kernel breakdown
  • Ray replica throughput (large-dataset path; not this spec)
  • Template search (HHsearch / HMMsearch); templates are attached from the spec's shipped structures only (see samples.md)
  • The demo tree examples/data/samples/ as a substitute dataset
  • DockQ / feature-tensor equivalence (make-data-pipeline)
  • Isolated same-features microbench of a single stack (Benchmark the Swap in docs/advanced/accelerate-custom-model.md) — the OSS feature path is e2e model.forward on OSS features, not a layer bench
  • Non-folding heads (boltz-2-affinity, embeddings, design, …) until this skill (or a sibling) is extended

Those are separate skills or a later revision.

Key gotchas

  • Launch each harness with its own python — $BIOIR_PYTHON or $OSS_PYTHON, never a bare python after Phase 0.
  • CUTEDSL_FORCE_CUBIN=1 on every BioIR process. JIT compile time makes forward latency unstable.
  • No PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, ever — it can slow forward (measurement.md).
  • CUDA 12 packages → CUDA 13. Install the cu13 counterpart of any cu12 pin (environment.md).
  • Protenix-v2 is Processor path. Never pass return_full_data=False; build_processor rejects it. Compare quality over seeds or the sample set because OSS adds Monte Carlo pair dropout that BioIR omits.
  • Seed goes on the feature-generator stage, not the tokenizer (docs/ref/api.md).
  • Paired A3Ms are per chain with equal row counts (docs/ref/api.md).
  • Let DockQ choose the chain mapping and record it.
  • Audit graph execution on every forward. Follow the counters and cache rules in measurement.md; the default model captures graphs without an optimize() call.
  • Sharing the GPU with another job invalidates the run.