scrna-embedding-audit
Score one or more single-cell embeddings stored in an h5ad (obsm) against cell-type labels and batch with scIB-style metrics, compute a PCA baseline from the same cells' raw counts in the same run, and state per metric whether each embedding scored higher, lower or not separably from that baseline.
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: scrna-embedding-auditdescription: >- Score one or more single-cell embeddings stored in an h5ad (obsm) against cell-type labels and batch with scIB-style metrics, compute a PCA baseline from the same cells' raw counts in the same run, and state per metric whether each embedding scored higher, lower or not separably from that baseline.license: MITmetadata: version: "0.1.0" author: Zhihao Wan domain: single-cell-transcriptomics tags: - single-cell - embedding - batch-integration - benchmark - scib - foundation-model inputs: - name: cells_h5ad type: file format: - h5ad description: AnnData with raw UMI counts in layers['counts'] or X, embeddings in obsm['X_*'], a cell-type column and optionally a batch column in obs required: false outputs: - name: report type: file format: - md description: Verdict table, full-data values, metric definitions, warnings - name: result type: file format: - json description: Every number in the report with provenance and definitions - name: tables type: file format: - csv description: metrics.csv, verdicts.csv, resamples.csv dependencies: python: ">=3.11" packages: - scanpy>=1.10 - anndata>=0.10 - numpy>=1.24 - pandas>=2.0 - scipy>=1.10 - scikit-learn>=1.3 - matplotlib>=3.7 demo_data: - path: examples/demo_spec.json description: Recipe for the offline synthetic demo (660 cells, 4 types, 2 donors, 3 count-derived embeddings) endpoints: cli: python skills/scrna-embedding-audit/scrna_embedding_audit.py --input {cells_h5ad} --output {output_dir} --labels-key cell_type openclaw: requires: bins: - python3 always: false emoji: "📏" homepage: https://github.com/ClawBio/ClawBio os: - darwin - linux install: - kind: pip package: scanpy trigger_keywords: - audit cell embedding - score embedding against PCA baseline - compare embeddings scIB metrics - batch integration metrics for obsm - is my scVI embedding better than PCA - evaluate foundation model embedding - iLISI cLISI silhouette for an h5ad📏 scRNA Embedding Audit
You are scRNA Embedding Audit, a ClawBio yardstick for single-cell embeddings. Your role is to score embeddings already stored in an h5ad against labels and batch, beside a PCA baseline computed from the same cells in the same run, and to say per metric what the run measured and nothing more.
Trigger
Fire this skill when the user says any of:
- "audit this embedding" / "score my embeddings against PCA"
- "is
X_scvi(orX_geneformer,X_harmony, anyobsmkey) better than PCA on my data" - "compute scIB metrics / iLISI / cLISI / silhouette / graph connectivity for an h5ad"
- "compare two embeddings for batch mixing and cell-type conservation"
- "evaluate a foundation-model cell embedding"
Do NOT fire when:
- the user wants to train an embedding (scVI, scANVI): route to
scrna-embedding - the user wants clustering, markers or cell-type annotation: route to
scrna-orchestrator - the user wants to run or benchmark a language model: route to
llm-biobank-bench - there is no embedding in
obsmyet; this skill scores embeddings, it does not make them (the PCA baseline is a reference, not an output)
Why This Exists
Independent benchmarks keep finding that zero-shot single-cell foundation models do not reliably beat simple baselines (Kedzierska et al., Genome Biology 2025). scrna-embedding reports three lightweight batch-mixing numbers for its own X_scvi, but nothing in ClawBio scores an arbitrary embedding for both biological conservation and batch mixing, and nothing puts a baseline next to the result in the same run.
- Without it: a user eyeballs a UMAP or trusts a model card.
- With it: ten metrics with pinned definitions, a baseline from the same cells, a random control for scale, and a per-metric verdict from a written half-sampling rule.
- Why ClawBio: every number is reproducible offline and traceable to a definition; the report says what was not measured.
Core Capabilities
- Score any
obsmembedding: scIB metrics re-implemented with numpy/scikit-learn and held to a fixture generated by scib-metrics 0.6.1. - Baseline in the same run: PCA from the raw counts of the scored cells (standard scanpy recipe), plus a seeded random control.
- Verdicts with a stated rule: 50 stratified half-samples;
higher/loweronly when every half-sample agrees in sign and the median difference exceeds 2 SD; otherwisenot separable,tiedornot evaluated. - Explicit abstention: missing batch, single label, degenerate embedding or dropped cells produce
nullwith a named reason, never a substituted number.
Scope
One skill, one task. This skill scores embeddings that already exist and reports how they compare with a PCA baseline on the user's cells. It does not train, integrate, cluster, annotate or write an h5ad. Label transfer with abstention is a separate planned skill; the single transfer metric here is a yardstick, not a labeller.
Input Formats
| Format | Extension | Required Fields | Example |
|---|---|---|---|
| AnnData | .h5ad | raw counts in layers["counts"] (preferred) or X; obsm["X_<name>"] embeddings (≥ 3 columns); obs[<labels-key>]; optional obs[<batch-key>] | integrated.h5ad from scrna-embedding with --counts-layer counts auto-detected |
Normalised or log-transformed matrices are refused (exit 2): the baseline needs counts and would log-transform them a second time. Any non-integer value in the count source is refused for the same reason, because a transformed matrix stored dense is mostly zeros and passes the shared loader's screen. Estimated or corrected counts (EM quantification, ambient-RNA correction) need --allow-non-integer-counts; the run then records their fraction and a warning.
Workflow
When the user asks to audit embeddings:
- Validate (prescriptive):
.h5adonly; counts screened byclawbio.common.scrna_io.load_count_adata, then non-integer values refused unless--allow-non-integer-counts; labels column must exist; cells with missing label/batch, excluded labels or zero counts are dropped and counted; fewer than 50 scorable cells → exit 2. - Select embeddings (prescriptive):
--embeddingsor, by default, everyobsm["X_*"]with ≥ 3 columns exceptX_umap,X_tsne,X_draw_graph_*,X_spatial,X_pca(listed in warnings). Above--max-cells(5000) one stratified subsample is scored. - Baseline (prescriptive): fresh AnnData from the scored cells' counts →
normalize_total(1e4)→log1p→ Seurat HVG (batch-aware when--batch-key) ifn_genes > --n-top-hvg→scale(max_value=10)→ arpack PCA in float64. Plus a seeded N(0, 1) control of the same dimension. - Score (prescriptive): the ten metrics below on the full scored data for every embedding, the baseline and the control, with one exact k-NN graph per embedding (self included, as scib-metrics expects).
- Resample (prescriptive): 50 stratified half-samples (label × batch strata, seeds
random_state + 1 + r); baseline metrics computed once per half-sample and shared. - Verdict (prescriptive): the rule in Algorithm / Methodology, per embedding and metric; counts per group.
- Report (flexible wording, fixed facts):
report.md,result.json, three CSV tables, one figure, reproducibility bundle. The agent may paraphrase the summary but must keep every verdict, number and caveat.
CLI Reference
# Standard usage
python skills/scrna-embedding-audit/scrna_embedding_audit.py \
--input cells.h5ad --output audit/ \
--embeddings X_scvi,X_geneformer --labels-key cell_type --batch-key donor
# Demo mode (synthetic data, offline)
python skills/scrna-embedding-audit/scrna_embedding_audit.py --demo --output /tmp/embedding_audit_demo
# Via ClawBio runner
python clawbio.py run scrna-embedding-audit --input cells.h5ad --output audit/ --labels-key cell_type --batch-key donor
python clawbio.py run scrna-embedding-audit --demo
Flags: --embeddings, --labels-key (required with --input), --batch-key, --exclude-labels, --counts-layer (default counts if present, else X), --allow-non-integer-counts, --n-top-hvg 2000, --n-pcs 50, --lisi-neighbors 90, --graph-neighbors 15, --transfer-k 15, --max-cells 5000, --n-resamples 50, --random-state 0, --expected-input-sha256, --overwrite. Bounds: lisi neighbours ≥ 3, resamples ≥ 10, max cells ≥ 50 (exit 2 otherwise). Through clawbio.py run, only these flags are forwarded; anything else is dropped silently by the runner's allowlist.
Demo
python clawbio.py run scrna-embedding-audit --demo
Expected output: a report on 660 synthetic cells (T cell, B cell, Monocyte in two donors; NK cell in one donor) with three count-derived embeddings: X_corrected (per-batch centred PCA) scores higher than the baseline on iLISI and graph connectivity, X_batchy (batch genes up-weighted) scores lower on every batch metric and most biology metrics, X_overcorrected (marker genes removed) scores lower on every biology metric. About 11 s single-threaded.
Algorithm / Methodology
Metric definitions follow scib-metrics 0.6.1 (Luecken et al. 2022; LISI from Korsunsky et al. 2019) and are checked against fixtures/scib_metrics_0.6.1_expected.json, generated by that release in an isolated Python 3.12 + JAX environment (fixtures/generate_scib_fixture.py). Agreement on the fixture: worst absolute difference 2.9e-7 (jax float32 upstream); test tolerance 1e-5. Higher is better for every metric; all lie in [0, 1] except ARI, which can be negative.
- Biological conservation (5):
silhouette_label(mean euclidean silhouette by label,(s+1)/2);isolated_labels(mean rescaled silhouette of the labels found in the fewest batches);nmi_kmeans,ari_kmeans(one KMeans withk = n_labels; scikit-learnn_init=10here, jaxn_init=1upstream, so not numerically comparable to a Benchmarker run);clisi(median per-cell LISI over labels on the 90-NN graph, perplexity 30, scaled(n_labels − med)/(n_labels − 1)). - Batch mixing (4):
bras(batch-removal-adapted silhouette, cosine, inter-batch term = mean distance to all other batches; the 0.6.1 Benchmarker default);silhouette_batch(legacy scIB batch ASW, euclidean, kept for continuity);ilisi(median LISI over batches on the 90-NN graph, scaled(med − 1)/(n_batches − 1));graph_connectivity(per label, largest strongly connected component of the directed 15-NN subgraph; computed without a batch key). - Cross-batch label transfer (1, this skill's own metric):
knn_transfer_accuracy, leave-one-batch-out distance-weighted kNN (k=15), balanced accuracy over labels present in the training batches; cells whose label never appears in the other batches are counted separately. - Not included (listed in
result.json): kBET, PCR comparison (0 for the baseline by construction), Leiden NMI/ARI. - Verdict rule (R2):
d_r = metric(embedding) − metric(baseline)on each of 50 half-samples;|d_r| ≤ 1e-5counts as 0.not evaluatedif any value is missing;tiedif everyd_ris 0 or both full-data values ≥ 0.999;higherif everyd_r > 0andmedian(d_r) > 2 · sd(d_r);lowersymmetric; otherwisenot separable. The factor 2, the tolerance and the ceiling are this skill's choices.
Key thresholds / parameters:
k = 90for LISI (perplexityfloor(k/3) = 30),k = 15for graph connectivity (source: scib-metrics 0.6.1Benchmarkerdefaults).- 50 half-samples, 2·SD factor, 1e-5 tolerance, 0.999 ceiling, 50-cell minimum, 5000-cell cap (source: this skill; see Gotchas for the calibration).
- Null calibration measured by this skill's author: two equally good embeddings (600 synthetic cells, 100 replicates) received a directional verdict on 0–5 % of metrics under this rule, against 15–43 % under the discarded 80 % × 20 sign-only rule (
workspacespike record referenced in the PR).
Example Queries
- "Is
X_scviinintegrated.h5adbetter than plain PCA oncell_typeacross donors?" - "Score
X_geneformerandX_scviagainst each other on my PBMC h5ad." - "Run scIB-style metrics for
obsm['X_harmony']with batchsample."
Example Output
Excerpt of report.md from python clawbio.py run scrna-embedding-audit --demo (2026-10-05, values as produced; the baseline and metric-definition header lines and the other table rows are omitted here):
# scrna-embedding-audit report
- **Generated**: 2026-10-05 23:25 UTC
- **Skill**: scrna-embedding-audit v0.1.0
- **Input**: synthetic demo (examples/demo_spec.json) (no measured data)
- **Cells**: 660 in file, 660 scored
- **Labels**: `cell_type` (4 unique)
- **Batch**: `donor` (2 unique)
## Summary
- `X_corrected` — biological conservation: higher than the PCA baseline on 3 of 5 metrics, lower on 0, not separable on 2, tied on 0, not evaluated on 0; batch mixing: higher than the PCA baseline on 2 of 4 metrics, lower on 0, not separable on 2, tied on 0, not evaluated on 0; cross-batch label transfer: higher than the PCA baseline on 0 of 1 metrics, lower on 0, not separable on 1, tied on 0, not evaluated on 0.
- `X_batchy` — biological conservation: higher than the PCA baseline on 0 of 5 metrics, lower on 4, not separable on 1, tied on 0, not evaluated on 0; batch mixing: higher than the PCA baseline on 0 of 4 metrics, lower on 4, not separable on 0, tied on 0, not evaluated on 0; cross-batch label transfer: higher than the PCA baseline on 0 of 1 metrics, lower on 1, not separable on 0, tied on 0, not evaluated on 0.
| embedding | metric | group | verdict | value | baseline | median diff | SD diff | frac > 0 | note |
|---|---|---|---|---:|---:|---:|---:|---:|---|
| `X_corrected` | silhouette_label | biological conservation | **higher** | 0.621 | 0.533 | 0.0880 | 0.0023 | 1.00 | |
| `X_corrected` | nmi_kmeans | biological conservation | **not separable** | 0.946 | 0.972 | -0.0217 | 0.0186 | 0.18 | differences change sign across half-samples |
| `X_corrected` | ilisi | batch mixing | **higher** | 0.867 | 0.514 | 0.3071 | 0.0365 | 1.00 | |
| `X_batchy` | ilisi | batch mixing | **lower** | 0.171 | 0.514 | -0.3325 | 0.0391 | 0.00 | |
Output Structure
output_directory/
├── report.md # Verdicts, full-data values, definitions, warnings, disclaimer
├── result.json # Every number with provenance; NaN written as null
├── tables/
│ ├── metrics.csv # embedding, metric, group, value, reason, provenance
│ ├── verdicts.csv # verdict and half-sample statistics per embedding and metric
│ └── resamples.csv # metric value in every half-sample
├── figures/
│ └── metrics_vs_baseline.png # full-data values, baseline and random control
└── reproducibility/
├── commands.sh # Replay command with the input hash guard
├── environment.yml # Pinned analysis packages
├── checksums.sha256 # Relative to the output directory
└── run_manifest.json # Parameters, input identity, environment, source hashes
Dependencies
Required (all present in uv sync --locked --all-extras; scanpy comes from the spatial extra):
scanpy≥ 1.10; baseline PCA recipeanndata; h5ad readingnumpy,scipy,scikit-learn; metrics, k-NN, KMeans, silhouettepandas; obs handlingmatplotlib; one figure
Not required at run time: scib-metrics and JAX (used once, in an isolated environment, to generate the committed fixture).
Gotchas
- You will want to score
X_umaporX_tsnebecause they are inobsm. Do not. Two-column layouts optimise local neighbourhoods for display; scored against a 50-PC baseline they give confident, meaningless verdicts. They are skipped by default and only scored when named in--embeddings. - You will want to read
higher on 3 of 5as "this embedding is better". Do not. The verdict describes this cell sample with these fixed embeddings. Half-sampling does not cover training seeds, donor-to-donor variation or the choice of k; the report says so in a fixed sentence. - You will want to compare
silhouette_batchhere withbatch_silhouettefromscrna-embedding. Do not.scrna-embeddingreports the raw silhouette by batch, where lower means better mixing; this skill reports scIB's1 − |s|averaged per label, where higher means better mixing. - You will want to call an scANVI or fine-tuned embedding the winner on biological conservation. Do not, without saying where the labels came from. An embedding trained on the labels it is scored against, or labels derived from one of the scored embeddings, scores circularly, and this skill cannot detect that.
- You will want to expect
graph_connectivity = 1for a single clean cluster. Do not. scib-metrics uses strongly connected components on the directed k-NN graph, so a boundary cell that nobody lists as a neighbour is its own component; one Gaussian blob of 100 cells scores about 0.99. - You will want to pass
--layeror--label-keythroughclawbio.py run. Do not. The runner forwards only the flags in its allowlist (--counts-layer,--labels-key, …); misspelt flags are dropped without an error. Run the script directly to get argparse's error. - You will want to add
--allow-non-integer-countsto get past the refusal on a normalisedX. Do not. The flag vouches that the values are counts. The baseline still runsnormalize_totalandlog1pon them, so a log-normalised matrix is transformed twice and the baseline is no longer the recipe the report states. Point--counts-layerat the raw counts; keep the flag for estimated or corrected counts. - You will want to read the random control as "another embedding". Do not.
control:randomis seeded noise that mixes batches perfectly and carries no biology; it is shown for scale and never enters a verdict. - You will want to read small
higher/lowercalls on a--max-cellsrun as method differences. Do not, when the embedding was fitted on more cells than are scored. The baseline is fitted on the scored subsample; an embedding fitted on the full dataset differs from it by fit population alone (on 44,721 PBMCs scored at 5,000, an identical recipe fitted on all cells differed by up to 0.09 on graph connectivity and got directional verdicts on 6 of 10 metrics;examples/public_validation.md). Raise--max-cellsor compare embeddings fitted on the same cells. - You will want to run this on a 64-core host with default BLAS threading. Do not expect it to be fast. The 660-cell demo takes about 11 s with
OMP_NUM_THREADS=1and about 40 s with every core spin-waiting; pin the thread count for large runs.
Safety
- Local-first: no network access anywhere in the skill;
--demois offline (tested with sockets blocked). - Disclaimer: every report ends with the ClawBio medical disclaimer ("ClawBio is a research and educational tool. It is not a medical device and does not provide clinical diagnoses. Consult a healthcare professional before making any medical decisions.").
- Audit trail: parameters, input hash, environment pins and source-file hashes in
reproducibility/run_manifest.json; checksums verify withcd <output_dir> && sha256sum -c reproducibility/checksums.sha256. - No hallucinated science: every metric definition names its source; the skill's own choices (2·SD, tolerance, ceiling, cell guards, transfer metric) are labelled as such in the report and
result.json.
Agent Boundary
The agent (LLM) dispatches and explains. The skill (Python) executes. The agent must not change the verdict rule, re-weight metrics into a composite, or report a superiority claim that the run did not measure.
Integration with Bio Orchestrator
Trigger conditions: the orchestrator routes here when:
- an h5ad with one or more
obsm["X_*"]embeddings and a cell-type column is to be evaluated - the user asks whether an integration or foundation-model embedding "beats PCA" or "mixes batches"
Chaining partners: this skill connects with:
scrna-embedding: itsintegrated.h5ad(X_scvi,layers["counts"]) is a direct input (--counts-layer countsis auto-detected)scrna-orchestrator: run the orchestrator with--use-repon whichever embedding this audit supports, with the audit's caveats attached- planned model-backed skills in this series hand their
obsm["X_<model>"]here instead of grading themselves
Maintenance
- Review cadence: when scib-metrics publishes a release that changes a default (as 0.6.x did for BRAS), regenerate the fixture in an isolated environment and record the version.
- Staleness signals: a new scanpy minor release (the HVG and scaling kernels changed between 1.11 and 1.12, so baseline values drift across versions; tests assert verdicts with margins, not pinned values); scikit-learn changing KMeans defaults.
- Deprecation: if ClawBio adopts scib-metrics as a dependency, this skill should call it directly and keep only the baseline, control and verdict logic.
Citations
- Luecken et al., Nat Methods 2022, 10.1038/s41592-021-01336-8; scIB metric definitions
- Korsunsky et al., Nat Methods 2019, 10.1038/s41592-019-0619-0; LISI
- scib-metrics 0.6.1; reference implementation used for the fixture
- Kedzierska et al., Genome Biology 2025, 10.1186/s13059-025-03574-x; zero-shot foundation-model embeddings versus baselines