Skip to content

Stage 11 — M3 Evaluation (Anti-Circular)

Pipeline position: 10 training → this stage → reporting (Stage 7)

Novel-site evaluation cannot reuse the M1-S/M2-S protocol (Stage 6), because evaluating a novel-site model against the annotation it trained on is circular. M3 has its own evaluator built around three principles — independent truth, held-out chromosomes, and per-gene (within-gene) ranking — with the novelty post-filter applied as part of the metric. The methodology is methods §6; this stage is how to run it.

Inputs → Outputs

Reads: a checkpoint (M3-S) or booster (M3-R), the D1/D2 truth sets, the annotation mask, and a per-gene .npz cache (recognizer) or the feature parquets (refiner). Writes: m3_eval_metrics.json + a comparison table (base vs each model on D1 / D1_hiconf / D2).

The two evaluators

Model Script How it scores
M3-S (recognizer) 13_evaluate_m3_novel.py dense per-position inference → post-filter → per-gene precision@k
M3-R (refiner) 15_evaluate_candidate_classifier.py rerank the candidate pool → same metric library

Both report per-gene precision@k / recall@k (k = 5/10/20) over the truth-containing genes on the held-out chromosomes (1, 3, 5, 7, 9), against three truth sets:

  • D1 — ENCODE long-read novel junctions (131,820 sites);
  • D1_hiconf — the ≥ 2-biosample subset (53,956);
  • D2 — held-out disease anchors (171 novel).

The novelty post-filter (subtract annotated sites before ranking) is built into the metric, so precision@k measures novelty recovery, not "is this a canonical site."

M3-S — recognizer eval (three modes)

13 separates the one GPU-dependent step (the feature cache) from the cheap local steps:

# 1. LOCAL — resolve the truth-containing test-gene universe (no pod)
python examples/meta_layer/13_evaluate_m3_novel.py --mode emit-universe

# 2. POD — build the shared per-gene .npz cache (bigWig; the only pod step)
python examples/meta_layer/13_evaluate_m3_novel.py --mode build-cache \
    --gene-list output/meta_layer/m3_eval_d1/eval_genes.txt \
    --cache-dir output/meta_layer/m3_eval_d1/gene_cache \
    --bigwig-cache /runpod-volume/bigwig_cache

# 3. LOCAL — score every model on the cache, report precision@k
python examples/meta_layer/13_evaluate_m3_novel.py --mode eval \
    --cache-dir output/meta_layer/m3_eval_d1/gene_cache

--mode eval scores base / M1-S / M2-S / M3-v1 / M3-v1-mm0 / M3-v1.1 / M3-anchor on the same candidate set, so every number is meta-vs-base on identical genes. It fails fast if base scores are missing (they would silently become a uniform prior and invalidate the eval) and loads each checkpoint via the config-type dispatcher load_meta_model (not the concat_fusion-hardcoded path in 08/09).

Two flags keep the comparison tractable:

  • --models base,M3-v1,M3-anchor scores a subset of DEFAULT_MODELS. Scoring every locally-present checkpoint over the 4,956-gene universe is ~15 h of CPU inference; a three-model subset is minutes.
  • --device mps runs forward inference on the Apple-Silicon GPU, ~13× faster than CPU (~65 min for the three-model comparison versus ~15 h on CPU).

The BatchNorm MPS warning is benign

On --device mps PyTorch warns about BatchNorm; that warning is autograd/training-only. Forward-only inference reproduces the published CPU M3-v1 numbers exactly, which doubles as a built-in reproducibility check.

M3-R — refiner eval

15 reranks the candidate pool by base score vs M3-R and scores both through the same metric library (_m3_novel_eval.py), by rasterizing per-candidate scores into the [L,3] array the recognizer path uses (with -inf fill so the precision@k denominator stays candidate-native):

python examples/meta_layer/15_evaluate_candidate_classifier.py \
    --universe output/meta_layer/m3_eval_d1/eval_genes.parquet

What the results say (Tier 0 / 1 / 2)

The evaluator was built as Tier 0 of an improvement ladder; the rungs and their honest outcomes:

Tier Experiment Verdict
0 Build the anti-circular eval, score M3-v1 M3-v1 is the best novel ranker (D1 P@5 0.335 vs base 0.277; D2 R@20 0.79 vs 0.51); multimodal adds little
1 Confirmed-only recognizer retrain No help — m3s_v1_1_confirmed marginally worse; label noise wasn't the ceiling
2 M3-R candidate refiner Honest negative — trains to AUC 0.90 but ties base; its edge is between-gene, not within-gene

The full tables and the within-vs-between-gene decomposition that explains Tier 2 are in results/m3_novel.md.

The reusable evaluation idea

Report per-gene precision@k against independent truth, not genome-wide PR-AUC against the training annotation. And when a model's global AUC and its per-gene precision@k disagree, decompose into within-gene vs between-gene — that decomposition is what turned M3-R from a mystery into a precise, reusable lesson (genome-averaged multimodal tracks are locus-level, not position-level).

Evaluating the disease-anchor fold (M3-anchor)

M3-anchor is M3 trained with the budgeted disease-anchor fold (Stage 10): output/meta_layer/m3s_anchorpos (best epoch 20, 5% budget), registered in DEFAULT_MODELS as M3-anchor. The question this stage answers is whether folding curated SF3B1 / ENCODE-KD cryptic sites in as positives helps the held-out disease truth without costing general novel recall.

python examples/meta_layer/13_evaluate_m3_novel.py --mode eval \
    --cache-dir output/meta_layer/m3_eval_d1/gene_cache \
    --output-dir output/meta_layer/m3_eval_d1 \
    --models base,M3-v1,M3-anchor --device mps

D1, ENCODE long-read novel (general recall):

Model P@5 R@20
base 0.277 0.315
M3-v1 0.335 0.367
M3-anchor 0.333 0.364

General novel recall holds (no regression): the fold does not trade away the D1 gain.

D2, held-out SF3B1 disease anchors (anti-circular by chromosome):

Model P@5 R@5 R@20
base 0.061 0.294 0.506
M3-v1 0.108 0.515 0.791
M3-anchor 0.120 0.571 0.859

M3-anchor improves across the board on the mechanism it was fed (R@20 +6.8 pts, R@5 +5.6 pts over M3-v1), and does so anti-circularly: the scored SF3B1 anchors sit on held-out chromosomes the fold never trained.

STMN2 / UNC13A within-gene rank (TDP-43, kept masked and untrained; a cross-mechanism transfer probe):

Site M3-v1 rank M3-anchor rank
STMN2 acceptor #4 #2
UNC13A donor #260 #187
UNC13A acceptor #660 #872

(of ~55K / ~87K novel candidates.) Because tdp43_cryptic is kept masked, the ALS panel is a genuine cross-mechanism probe (SF3B1-learning was never shown these sites). The result is mixed (2 of 3 improved): transfer to the ALS regime is only partial.

What the anchor fold does, and does not, do

The budgeted fold works within-mechanism (SF3B1) without hurting general recall (D1). Cross-mechanism transfer to the weak, perturbation-gated TDP-43 regime is weak, which is M4's domain (conditional, regulator-driven), not M3's. M3-anchor is not yet promoted: this is a single split and the decision is pending. Full write-up: results/m3_novel.md.

Inspect the ranker interactively — the Novel Site Explorer

The aggregate precision@k numbers say M3 ranks well; the Novel Site Explorer lets you see it per gene. Launch the Bio Lab UI and open a gene:

conda run -n agentic-spliceai python -m server.bio.app   # http://localhost:8005/novel/TPR

For a gene it shows M3's top-k candidate unannotated sites with the novelty post-filter applied, each row carrying its M3 score, the base model's score and rank at the same position, the splice dinucleotide, and independent evidence badges — ENCODE long-read support (D1, with biosample count) and held-out disease anchors (D2, with mechanism). SpliceVault/positives_pooled is deliberately not shown as evidence: it is M3's training pool, so it would be circular.

The serving universe is the same held-out set this stage evaluates (test chromosomes ⅓/5/7/9), so every inspectable gene is one M3 never trained on. TPR is a good first look — both of its SF3B1-cryptic anchors land in the top 5, at base scores of ~0.09–0.13.

Implementation: server/bio/m3_inference.py, sharing the ranking primitives in splice_engine/eval/novel_site_ranking.py with this stage's harness, so the UI and the eval rank identically by construction.

→ Reporting: roll these into the results pages as in Stage 7 — Reporting.