← LeaderboardThe Skill Leaderboard

scrna-embedding-audit

Score one or more single-cell embeddings stored in an h5ad (obsm) against cell-type labels and batch with scIB-style metrics, compute a PCA baseline from the same cells' raw counts in the same run, and state per metric whether each embedding scored higher, lower or not separably from that baseline.

Published by ClawBio ↗ClawBio/ClawBio

Try your first task

A source-grounded starter task has not been verified for this skill yet.

SKILL.md

name: scrna-embedding-auditdescription: >-  Score one or more single-cell embeddings stored in an h5ad (obsm) against  cell-type labels and batch with scIB-style metrics, compute a PCA baseline  from the same cells' raw counts in the same run, and state per metric whether  each embedding scored higher, lower or not separably from that baseline.license: MITmetadata:   version: "0.1.0"  author: Zhihao Wan  domain: single-cell-transcriptomics  tags:     - single-cell    - embedding    - batch-integration    - benchmark    - scib    - foundation-model  inputs:     - name: cells_h5ad      type: file      format:         - h5ad      description: AnnData with raw UMI counts in layers['counts'] or X, embeddings in obsm['X_*'], a cell-type column and optionally a batch column in obs      required: false  outputs:     - name: report      type: file      format:         - md      description: Verdict table, full-data values, metric definitions, warnings    - name: result      type: file      format:         - json      description: Every number in the report with provenance and definitions    - name: tables      type: file      format:         - csv      description: metrics.csv, verdicts.csv, resamples.csv  dependencies:     python: ">=3.11"    packages:       - scanpy>=1.10      - anndata>=0.10      - numpy>=1.24      - pandas>=2.0      - scipy>=1.10      - scikit-learn>=1.3      - matplotlib>=3.7  demo_data:     - path: examples/demo_spec.json      description: Recipe for the offline synthetic demo (660 cells, 4 types, 2 donors, 3 count-derived embeddings)  endpoints:     cli: python skills/scrna-embedding-audit/scrna_embedding_audit.py --input {cells_h5ad} --output {output_dir} --labels-key cell_type  openclaw:     requires:       bins:         - python3    always: false    emoji: "📏"    homepage: https://github.com/ClawBio/ClawBio    os:       - darwin      - linux    install:       - kind: pip        package: scanpy    trigger_keywords:       - audit cell embedding      - score embedding against PCA baseline      - compare embeddings scIB metrics      - batch integration metrics for obsm      - is my scVI embedding better than PCA      - evaluate foundation model embedding      - iLISI cLISI silhouette for an h5ad

📏 scRNA Embedding Audit

You are scRNA Embedding Audit, a ClawBio yardstick for single-cell embeddings. Your role is to score embeddings already stored in an h5ad against labels and batch, beside a PCA baseline computed from the same cells in the same run, and to say per metric what the run measured and nothing more.

Trigger

Fire this skill when the user says any of:

  • "audit this embedding" / "score my embeddings against PCA"
  • "is X_scvi (or X_geneformer, X_harmony, any obsm key) better than PCA on my data"
  • "compute scIB metrics / iLISI / cLISI / silhouette / graph connectivity for an h5ad"
  • "compare two embeddings for batch mixing and cell-type conservation"
  • "evaluate a foundation-model cell embedding"

Do NOT fire when:

  • the user wants to train an embedding (scVI, scANVI): route to scrna-embedding
  • the user wants clustering, markers or cell-type annotation: route to scrna-orchestrator
  • the user wants to run or benchmark a language model: route to llm-biobank-bench
  • there is no embedding in obsm yet; this skill scores embeddings, it does not make them (the PCA baseline is a reference, not an output)

Why This Exists

Independent benchmarks keep finding that zero-shot single-cell foundation models do not reliably beat simple baselines (Kedzierska et al., Genome Biology 2025). scrna-embedding reports three lightweight batch-mixing numbers for its own X_scvi, but nothing in ClawBio scores an arbitrary embedding for both biological conservation and batch mixing, and nothing puts a baseline next to the result in the same run.

  • Without it: a user eyeballs a UMAP or trusts a model card.
  • With it: ten metrics with pinned definitions, a baseline from the same cells, a random control for scale, and a per-metric verdict from a written half-sampling rule.
  • Why ClawBio: every number is reproducible offline and traceable to a definition; the report says what was not measured.

Core Capabilities

  1. Score any obsm embedding: scIB metrics re-implemented with numpy/scikit-learn and held to a fixture generated by scib-metrics 0.6.1.
  2. Baseline in the same run: PCA from the raw counts of the scored cells (standard scanpy recipe), plus a seeded random control.
  3. Verdicts with a stated rule: 50 stratified half-samples; higher/lower only when every half-sample agrees in sign and the median difference exceeds 2 SD; otherwise not separable, tied or not evaluated.
  4. Explicit abstention: missing batch, single label, degenerate embedding or dropped cells produce null with a named reason, never a substituted number.

Scope

One skill, one task. This skill scores embeddings that already exist and reports how they compare with a PCA baseline on the user's cells. It does not train, integrate, cluster, annotate or write an h5ad. Label transfer with abstention is a separate planned skill; the single transfer metric here is a yardstick, not a labeller.

Input Formats

FormatExtensionRequired FieldsExample
AnnData.h5adraw counts in layers["counts"] (preferred) or X; obsm["X_<name>"] embeddings (≥ 3 columns); obs[<labels-key>]; optional obs[<batch-key>]integrated.h5ad from scrna-embedding with --counts-layer counts auto-detected

Normalised or log-transformed matrices are refused (exit 2): the baseline needs counts and would log-transform them a second time. Any non-integer value in the count source is refused for the same reason, because a transformed matrix stored dense is mostly zeros and passes the shared loader's screen. Estimated or corrected counts (EM quantification, ambient-RNA correction) need --allow-non-integer-counts; the run then records their fraction and a warning.

Workflow

When the user asks to audit embeddings:

  1. Validate (prescriptive): .h5ad only; counts screened by clawbio.common.scrna_io.load_count_adata, then non-integer values refused unless --allow-non-integer-counts; labels column must exist; cells with missing label/batch, excluded labels or zero counts are dropped and counted; fewer than 50 scorable cells → exit 2.
  2. Select embeddings (prescriptive): --embeddings or, by default, every obsm["X_*"] with ≥ 3 columns except X_umap, X_tsne, X_draw_graph_*, X_spatial, X_pca (listed in warnings). Above --max-cells (5000) one stratified subsample is scored.
  3. Baseline (prescriptive): fresh AnnData from the scored cells' counts → normalize_total(1e4) → log1p → Seurat HVG (batch-aware when --batch-key) if n_genes > --n-top-hvg → scale(max_value=10) → arpack PCA in float64. Plus a seeded N(0, 1) control of the same dimension.
  4. Score (prescriptive): the ten metrics below on the full scored data for every embedding, the baseline and the control, with one exact k-NN graph per embedding (self included, as scib-metrics expects).
  5. Resample (prescriptive): 50 stratified half-samples (label × batch strata, seeds random_state + 1 + r); baseline metrics computed once per half-sample and shared.
  6. Verdict (prescriptive): the rule in Algorithm / Methodology, per embedding and metric; counts per group.
  7. Report (flexible wording, fixed facts): report.md, result.json, three CSV tables, one figure, reproducibility bundle. The agent may paraphrase the summary but must keep every verdict, number and caveat.

CLI Reference

# Standard usage
python skills/scrna-embedding-audit/scrna_embedding_audit.py \
  --input cells.h5ad --output audit/ \
  --embeddings X_scvi,X_geneformer --labels-key cell_type --batch-key donor

# Demo mode (synthetic data, offline)
python skills/scrna-embedding-audit/scrna_embedding_audit.py --demo --output /tmp/embedding_audit_demo

# Via ClawBio runner
python clawbio.py run scrna-embedding-audit --input cells.h5ad --output audit/ --labels-key cell_type --batch-key donor
python clawbio.py run scrna-embedding-audit --demo

Flags: --embeddings, --labels-key (required with --input), --batch-key, --exclude-labels, --counts-layer (default counts if present, else X), --allow-non-integer-counts, --n-top-hvg 2000, --n-pcs 50, --lisi-neighbors 90, --graph-neighbors 15, --transfer-k 15, --max-cells 5000, --n-resamples 50, --random-state 0, --expected-input-sha256, --overwrite. Bounds: lisi neighbours ≥ 3, resamples ≥ 10, max cells ≥ 50 (exit 2 otherwise). Through clawbio.py run, only these flags are forwarded; anything else is dropped silently by the runner's allowlist.

Demo

python clawbio.py run scrna-embedding-audit --demo

Expected output: a report on 660 synthetic cells (T cell, B cell, Monocyte in two donors; NK cell in one donor) with three count-derived embeddings: X_corrected (per-batch centred PCA) scores higher than the baseline on iLISI and graph connectivity, X_batchy (batch genes up-weighted) scores lower on every batch metric and most biology metrics, X_overcorrected (marker genes removed) scores lower on every biology metric. About 11 s single-threaded.

Algorithm / Methodology

Metric definitions follow scib-metrics 0.6.1 (Luecken et al. 2022; LISI from Korsunsky et al. 2019) and are checked against fixtures/scib_metrics_0.6.1_expected.json, generated by that release in an isolated Python 3.12 + JAX environment (fixtures/generate_scib_fixture.py). Agreement on the fixture: worst absolute difference 2.9e-7 (jax float32 upstream); test tolerance 1e-5. Higher is better for every metric; all lie in [0, 1] except ARI, which can be negative.

  1. Biological conservation (5): silhouette_label (mean euclidean silhouette by label, (s+1)/2); isolated_labels (mean rescaled silhouette of the labels found in the fewest batches); nmi_kmeans, ari_kmeans (one KMeans with k = n_labels; scikit-learn n_init=10 here, jax n_init=1 upstream, so not numerically comparable to a Benchmarker run); clisi (median per-cell LISI over labels on the 90-NN graph, perplexity 30, scaled (n_labels − med)/(n_labels − 1)).
  2. Batch mixing (4): bras (batch-removal-adapted silhouette, cosine, inter-batch term = mean distance to all other batches; the 0.6.1 Benchmarker default); silhouette_batch (legacy scIB batch ASW, euclidean, kept for continuity); ilisi (median LISI over batches on the 90-NN graph, scaled (med − 1)/(n_batches − 1)); graph_connectivity (per label, largest strongly connected component of the directed 15-NN subgraph; computed without a batch key).
  3. Cross-batch label transfer (1, this skill's own metric): knn_transfer_accuracy, leave-one-batch-out distance-weighted kNN (k=15), balanced accuracy over labels present in the training batches; cells whose label never appears in the other batches are counted separately.
  4. Not included (listed in result.json): kBET, PCR comparison (0 for the baseline by construction), Leiden NMI/ARI.
  5. Verdict rule (R2): d_r = metric(embedding) − metric(baseline) on each of 50 half-samples; |d_r| ≤ 1e-5 counts as 0. not evaluated if any value is missing; tied if every d_r is 0 or both full-data values ≥ 0.999; higher if every d_r > 0 and median(d_r) > 2 · sd(d_r); lower symmetric; otherwise not separable. The factor 2, the tolerance and the ceiling are this skill's choices.

Key thresholds / parameters:

  • k = 90 for LISI (perplexity floor(k/3) = 30), k = 15 for graph connectivity (source: scib-metrics 0.6.1 Benchmarker defaults).
  • 50 half-samples, 2·SD factor, 1e-5 tolerance, 0.999 ceiling, 50-cell minimum, 5000-cell cap (source: this skill; see Gotchas for the calibration).
  • Null calibration measured by this skill's author: two equally good embeddings (600 synthetic cells, 100 replicates) received a directional verdict on 0–5 % of metrics under this rule, against 15–43 % under the discarded 80 % × 20 sign-only rule (workspace spike record referenced in the PR).

Example Queries

  • "Is X_scvi in integrated.h5ad better than plain PCA on cell_type across donors?"
  • "Score X_geneformer and X_scvi against each other on my PBMC h5ad."
  • "Run scIB-style metrics for obsm['X_harmony'] with batch sample."

Example Output

Excerpt of report.md from python clawbio.py run scrna-embedding-audit --demo (2026-10-05, values as produced; the baseline and metric-definition header lines and the other table rows are omitted here):

# scrna-embedding-audit report

- **Generated**: 2026-10-05 23:25 UTC
- **Skill**: scrna-embedding-audit v0.1.0
- **Input**: synthetic demo (examples/demo_spec.json) (no measured data)
- **Cells**: 660 in file, 660 scored
- **Labels**: `cell_type` (4 unique)
- **Batch**: `donor` (2 unique)

## Summary

- `X_corrected` — biological conservation: higher than the PCA baseline on 3 of 5 metrics, lower on 0, not separable on 2, tied on 0, not evaluated on 0; batch mixing: higher than the PCA baseline on 2 of 4 metrics, lower on 0, not separable on 2, tied on 0, not evaluated on 0; cross-batch label transfer: higher than the PCA baseline on 0 of 1 metrics, lower on 0, not separable on 1, tied on 0, not evaluated on 0.
- `X_batchy` — biological conservation: higher than the PCA baseline on 0 of 5 metrics, lower on 4, not separable on 1, tied on 0, not evaluated on 0; batch mixing: higher than the PCA baseline on 0 of 4 metrics, lower on 4, not separable on 0, tied on 0, not evaluated on 0; cross-batch label transfer: higher than the PCA baseline on 0 of 1 metrics, lower on 1, not separable on 0, tied on 0, not evaluated on 0.

| embedding | metric | group | verdict | value | baseline | median diff | SD diff | frac > 0 | note |
|---|---|---|---|---:|---:|---:|---:|---:|---|
| `X_corrected` | silhouette_label | biological conservation | **higher** | 0.621 | 0.533 | 0.0880 | 0.0023 | 1.00 |  |
| `X_corrected` | nmi_kmeans | biological conservation | **not separable** | 0.946 | 0.972 | -0.0217 | 0.0186 | 0.18 | differences change sign across half-samples |
| `X_corrected` | ilisi | batch mixing | **higher** | 0.867 | 0.514 | 0.3071 | 0.0365 | 1.00 |  |
| `X_batchy` | ilisi | batch mixing | **lower** | 0.171 | 0.514 | -0.3325 | 0.0391 | 0.00 |  |

Output Structure

output_directory/
├── report.md                      # Verdicts, full-data values, definitions, warnings, disclaimer
├── result.json                    # Every number with provenance; NaN written as null
├── tables/
│   ├── metrics.csv                # embedding, metric, group, value, reason, provenance
│   ├── verdicts.csv               # verdict and half-sample statistics per embedding and metric
│   └── resamples.csv              # metric value in every half-sample
├── figures/
│   └── metrics_vs_baseline.png    # full-data values, baseline and random control
└── reproducibility/
    ├── commands.sh                # Replay command with the input hash guard
    ├── environment.yml            # Pinned analysis packages
    ├── checksums.sha256           # Relative to the output directory
    └── run_manifest.json          # Parameters, input identity, environment, source hashes

Dependencies

Required (all present in uv sync --locked --all-extras; scanpy comes from the spatial extra):

  • scanpy ≥ 1.10; baseline PCA recipe
  • anndata; h5ad reading
  • numpy, scipy, scikit-learn; metrics, k-NN, KMeans, silhouette
  • pandas; obs handling
  • matplotlib; one figure

Not required at run time: scib-metrics and JAX (used once, in an isolated environment, to generate the committed fixture).

Gotchas

  • You will want to score X_umap or X_tsne because they are in obsm. Do not. Two-column layouts optimise local neighbourhoods for display; scored against a 50-PC baseline they give confident, meaningless verdicts. They are skipped by default and only scored when named in --embeddings.
  • You will want to read higher on 3 of 5 as "this embedding is better". Do not. The verdict describes this cell sample with these fixed embeddings. Half-sampling does not cover training seeds, donor-to-donor variation or the choice of k; the report says so in a fixed sentence.
  • You will want to compare silhouette_batch here with batch_silhouette from scrna-embedding. Do not. scrna-embedding reports the raw silhouette by batch, where lower means better mixing; this skill reports scIB's 1 − |s| averaged per label, where higher means better mixing.
  • You will want to call an scANVI or fine-tuned embedding the winner on biological conservation. Do not, without saying where the labels came from. An embedding trained on the labels it is scored against, or labels derived from one of the scored embeddings, scores circularly, and this skill cannot detect that.
  • You will want to expect graph_connectivity = 1 for a single clean cluster. Do not. scib-metrics uses strongly connected components on the directed k-NN graph, so a boundary cell that nobody lists as a neighbour is its own component; one Gaussian blob of 100 cells scores about 0.99.
  • You will want to pass --layer or --label-key through clawbio.py run. Do not. The runner forwards only the flags in its allowlist (--counts-layer, --labels-key, …); misspelt flags are dropped without an error. Run the script directly to get argparse's error.
  • You will want to add --allow-non-integer-counts to get past the refusal on a normalised X. Do not. The flag vouches that the values are counts. The baseline still runs normalize_total and log1p on them, so a log-normalised matrix is transformed twice and the baseline is no longer the recipe the report states. Point --counts-layer at the raw counts; keep the flag for estimated or corrected counts.
  • You will want to read the random control as "another embedding". Do not. control:random is seeded noise that mixes batches perfectly and carries no biology; it is shown for scale and never enters a verdict.
  • You will want to read small higher/lower calls on a --max-cells run as method differences. Do not, when the embedding was fitted on more cells than are scored. The baseline is fitted on the scored subsample; an embedding fitted on the full dataset differs from it by fit population alone (on 44,721 PBMCs scored at 5,000, an identical recipe fitted on all cells differed by up to 0.09 on graph connectivity and got directional verdicts on 6 of 10 metrics; examples/public_validation.md). Raise --max-cells or compare embeddings fitted on the same cells.
  • You will want to run this on a 64-core host with default BLAS threading. Do not expect it to be fast. The 660-cell demo takes about 11 s with OMP_NUM_THREADS=1 and about 40 s with every core spin-waiting; pin the thread count for large runs.

Safety

  • Local-first: no network access anywhere in the skill; --demo is offline (tested with sockets blocked).
  • Disclaimer: every report ends with the ClawBio medical disclaimer ("ClawBio is a research and educational tool. It is not a medical device and does not provide clinical diagnoses. Consult a healthcare professional before making any medical decisions.").
  • Audit trail: parameters, input hash, environment pins and source-file hashes in reproducibility/run_manifest.json; checksums verify with cd <output_dir> && sha256sum -c reproducibility/checksums.sha256.
  • No hallucinated science: every metric definition names its source; the skill's own choices (2·SD, tolerance, ceiling, cell guards, transfer metric) are labelled as such in the report and result.json.

Agent Boundary

The agent (LLM) dispatches and explains. The skill (Python) executes. The agent must not change the verdict rule, re-weight metrics into a composite, or report a superiority claim that the run did not measure.

Integration with Bio Orchestrator

Trigger conditions: the orchestrator routes here when:

  • an h5ad with one or more obsm["X_*"] embeddings and a cell-type column is to be evaluated
  • the user asks whether an integration or foundation-model embedding "beats PCA" or "mixes batches"

Chaining partners: this skill connects with:

  • scrna-embedding: its integrated.h5ad (X_scvi, layers["counts"]) is a direct input (--counts-layer counts is auto-detected)
  • scrna-orchestrator: run the orchestrator with --use-rep on whichever embedding this audit supports, with the audit's caveats attached
  • planned model-backed skills in this series hand their obsm["X_<model>"] here instead of grading themselves

Maintenance

  • Review cadence: when scib-metrics publishes a release that changes a default (as 0.6.x did for BRAS), regenerate the fixture in an isolated environment and record the version.
  • Staleness signals: a new scanpy minor release (the HVG and scaling kernels changed between 1.11 and 1.12, so baseline values drift across versions; tests assert verdicts with margins, not pinned values); scikit-learn changing KMeans defaults.
  • Deprecation: if ClawBio adopts scib-metrics as a dependency, this skill should call it directly and keep only the baseline, control and verdict logic.

Citations