Citation integrity in evaluation

A lawyer reading a model output cares whether a named case exists, whether it is the right case, whether the quote is faithful, and whether the proposition follows. Those are different questions. Collapsing them into “accuracy” produces a score that looks decisive and explains almost nothing when something goes wrong.

England & Wales practice is particularly exposed. Neutral citations, party names, and pinpoints are how counsel and the court check a claim. An invented authority is a different risk from a paraphrase that quietly changes the holding. Evaluation that cannot tell them apart cannot guide product work or institutional adoption.

General-purpose chat benchmarks rarely isolate these modes. Legal marketing pages often do not either. The result is confidence without a checkable trail.

This report concerns evaluation of model outputs against England & Wales legal sources — primarily UK judgments held in a stable, stored corpus. It is about how we score claims that cite, quote, or attribute propositions to those sources.

It is not a product feature specification. Verify runs citation and quote checks on matter documents in the workspace; this report describes the evaluation frame behind that work. Search over Obiter’s stored UK judgment corpus is the shipping grounding surface; legislation search is on the roadmap as its own slice.

It is also not a claim about private matter files. Matter uploads stay organisation-scoped. They are not folded into the open legal index, and they are not used here as evaluation gold without an explicit, separate protocol.

We treat the following classes as distinct. A single output may exhibit more than one. Scoring must record them separately rather than averaging them into a headline rate.

The taxonomy is deliberately operational. Each class should be adjudicable against a stored source record and a frozen gold item. Ambiguous edge cases belong in adjudication notes, not in a silent merge of categories.

Axes are the published score dimensions once a suite is frozen. They map to the failure modes above but are phrased as positive checks so a suite can report pass rates without implying a single composite.

We refuse one headline accuracy number for citation integrity. If a downstream consumer needs a summary, it should be an explicit, named composite with a formula — not a marketing average of unlike errors.

A suite starts as an internal gold set: prompts or claim bundles, expected authorities, expected quote spans where relevant, and proposition labels. Items are versioned. Changing an item bumps the suite revision.

Replay runs the system under a fixed retrieval and generation configuration against that gold set. Outputs are scored per axis. Human adjudication covers borderline form errors and proposition support; the adjudication rubric ships with the suite when it freezes.

“Frozen” means: item IDs stable, scoring rules published, corpus snapshot or identifier policy declared, and manifests available so another party can replay. Until then, internal runs are drafts. They are not comparable public scores.

Obiter Research publishes methodology when it is ready to defend. Comparable scores land only with a frozen suite and public manifests. Partial internal results are not leaderboards.

This report is methodology. The citation resolution suite noted on the Benchmark page remains internal at v0.3. Public methodology and manifests for that suite will ship when it freezes — not before.

If a published score later needs revision because of a corpus or rubric fix, we bump the suite version and state what changed. Quiet rescoring of a named public figure is not acceptable.

The same failure modes that break evaluation also break trust in a workspace. Verify checks citations, quotations, and authorities against stored sources on matter documents; this report describes the frame those checks are scored under.

Search already serves UK judgments from Obiter’s own storage inside the signed-in workspace, and will in public search at /search. Stable stored case pages are the grounding surface evaluation assumes. Without reproducible retrieval, citation integrity scoring has nothing solid to check against.

Redact and Matters sit on the other side of the trust boundary: private documents and disclosure review. They are out of scope for this open-corpus evaluation frame, except to restate that matter content is not mixed into the public index.