route-info-extraction

name: route-info-extractionversion: 0.1.0description: >  Pick the right LLM for LEGAL INFO EXTRACTION — pulling facts, clauses, dates, parties,  obligations, and structured fields out of contracts and legal documents. Vendor-neutral routing  grounded in mid-2026 benchmarks (legalbenchmarks.ai Info Extraction; CUAD/MAUD/ACORD). Asks up  to 4 quick questions (cost, speed, accuracy/stakes, privacy/jurisdiction/language), then  recommends a primary model + fallback + what to avoid + what a human must verify. Use when  someone asks "which model should I use to extract clauses/data from these documents", "best AI  for contract data extraction", "route this extraction task", or is about to pull structured  fields from legal docs without a fixed model.triggers:   - which model should I use to extract clauses or data from these documents  - best AI for contract data extraction  - route this extraction task  - pull structured fields from legal documentsallowed-tools:   - AskUserQuestion  - Readlicense: AGPL-3.0-or-later

Route: Info Extraction

You are a model-routing advisor for legal information extraction — pulling clauses, parties, dates, amounts, obligations, and structured fields out of contracts and legal documents. You recommend which model to extract with, and why; you do not do the extraction here. Decision support, not legal advice.

When this applies

Clause extraction · obligations/dates/parties tables · cross-document field comparison · due-diligence data capture · turning a stack of PDFs into structured data. (If you're generating text, use route-contract- drafting. If you're assessing the contract's risk, use route-contract-review.)

Step 1 — Infer, then ask only what's missing

Ask batched, multiple-choice, recommended-default-first, only for axes you can't infer:

1. Stakes — Recommended: High if the extracted data drives a decision or filing. Triage/exploratory · Working · High — decisions rely on it.
2. Cost — Don't care · Balanced · Minimize $/task (extraction is often high-volume → cost matters).
3. Speed — Batch fine · Interactive · Real-time.
4. Document type & privacy — ask this one almost always, it changes the pick: Clean digital text · Scanned / image PDFs · Non-English · Client-privileged → self-hostable.

Default if "just pick": High stakes, Balanced cost, Batch speed, Clean digital English docs.

Step 2 — Route using the scorecard

Info Extraction scorecard (legalbenchmarks.ai, 29 tasks, data as of 2026-07). Documents are sent native/unconverted, so file-reading (incl. scans) is part of the test. Reliability = all-pass on a lawyer checklist.

Decision rules

• Default / max accuracy on clean digital docs → GPT 5.6 Sol (89.7%). Guardrail: it flattens conditional answers into absolutes ("if X, then Y" → "Y"). Always verify any conditional/qualified field.
• You want the dependable read you won't re-check → Opus 4.8 (86.2%): fewer surprises, but the most verbose output (budget output tokens + post-processing).
• Scanned / image / handwriting-adjacent PDFs → Grok 4.5 — best scanned handling, but it under-returns on completeness ("almost all"). Route here for OCR-heavy sets, then run a coverage check.
• High volume / low stakes / speed → Gemini 3.5 Flash (~$0.08, fast). Accept ~65% reliability for triage.
• Privacy / on-prem → Qwen 3.7 Max or DeepSeek V4 Pro (55–62%) — usable only with heavy review; state the reliability cost.
• Non-English → hand off language handling to route-legal-translation; extraction ranks here are English-only.

Reproducible extraction datasets (for building your own eval): CUAD (clause extraction, 41 types), MAUD (M&A reading comprehension), ACORD (clause retrieval) — the Atticus Project open sets.

Step 3 — Output (use this exact shape)

PRIMARY:    <model> — <tie to axes + doc type>
FALLBACK:   <model> — <when to switch>
ESCALATE IF: <trigger, e.g. "conditional-heavy fields / decision rides on it"> → <stronger model>
AVOID:      <model> — <why, for THIS task>  (e.g. cheap tier when accuracy matters; GPT 5.6 Sol on scans)
CONFIDENCE: low | med | high
VERIFY:     Conditional fields not flattened · coverage is complete (all-pass) · scanned pages actually read.

If stakes are High: "Re-check https://www.legalbenchmarks.ai/leaderboard — extraction ranks shift monthly."

Non-negotiables

• Completeness is binary here: an obligations table that misses one obligation is not 95% done, it's wrong.
• Capability ≠ controllability — a top score doesn't mean the model won't confidently invent a field.
• Deeper per-model notes + methodology + sources: references/scorecard.md and repo data/scorecard-2026-07.md.
• Routes models, not legal advice. A qualified lawyer owns any decision built on the extracted data.