How answers are verified, and how we measure it
Legal AI tools have been publicly caught claiming accuracy they could not substantiate. This page is the opposite approach: the exact mechanism, the exact test method, the exact numbers with dates, and a plain statement of what none of it guarantees. The evaluation harnesses ship in the open source repository so anyone can rerun them.
The mechanism
Before an answer is shown, code (not the AI) checks every quoted passage character for character against the retrieved text of the cited document. A claim whose quote does not match is rejected and counted, never displayed. Citations carry the exact page and passage offsets; transcript line numbers are computed from the verified quote's position. If the documents do not support an answer, the system says "I could not find this in the documents" rather than guessing, after automatically retrying once with a wider search.
The evaluation, as of July 7, 2026
A golden set of 72 questions over a synthetic legal corpus: 63 questions whose answers exist at a known page and passage, and 9 questions whose answers deliberately do not exist anywhere in the corpus. Grading is strict: a present-fact question passes only if the answer cites the right document and page with a mechanically verified quote; an absent-fact question passes only if the system refuses.
| Measure | Result |
|---|---|
| Present-fact questions correctly answered with a verified page citation | 63 of 63 (one credited on an alternate page where the same fact appears on two pages) |
| Fabricated citations displayed | 0, across every recorded run since the harness was built |
| Absent-fact questions correctly refused | 9 of 9 |
| Cross-matter isolation checks (right client's document, never another client's) | 2 of 2 |
A second, larger evaluation tests retrieval at scale: 5,114 indexed passages across 50 synthetic client matters of realistic size, with 50 additional questions spanning the query shapes attorneys actually use (invoice and case numbers, dollar amounts, statute citations, rare names, defined terms, colloquial paraphrases, near-identical contract twins). In the July 7, 2026 run, retrieval placed the correct passage in the top five for 100 percent of those query classes, and 98 percent of the golden questions, under the exact configuration the product ships.
Error taxonomy we grade against
- Incorrect: the answer states something false. Caught by the strict grading above.
- Misgrounded: the statement is right but the cited source does not support it. This is the failure mode the mechanical quote check exists to stop.
- Unsupported: the model asserted something it could not quote. Rejected before display and counted in the run records.
- False refusal: the answer existed but the system said it could not find it. Tracked separately; the July 7 run recovered the one known false refusal via the wider second-pass search. Refusal behavior is reported, never hidden inside an accuracy number.
What these numbers do not mean. They are measured on synthetic legal documents, not your documents; hard cases in your files can behave differently. They measure whether quotes and citations are faithful, not whether any statement is legally correct, current, or complete. A verified answer can still miss context elsewhere in the record. docuchat locates and summarizes; the attorney verifies and judges. It is not an AI lawyer.
Run records and harnesses: eval/SCALE_EVAL.md, eval/LATENCY.md, and eval/TEST_PLAN.md in the repository. Product version: run-from-source, July 7, 2026. This page changes only when a new recorded run changes the numbers.