How to read a legal-AI benchmark without being fooled
2026-10-18 · Models
Vendor benchmarks are marketing. A one-afternoon test on your own matters beats any leaderboard.
Short answer
Most legal-AI benchmarks are run by the vendor, on tasks they chose, scored by a method they control. Treat them as a starting point, not evidence. Build a small test set from your own work — a handful of real tasks with known-good answers and a trap or two — and score each model on accuracy, citation behaviour and how it handles uncertainty. What matters is not which model tops a leaderboard but which one is reliable on your matters.
Why vendor benchmarks mislead
- The vendor picks the tasks.
- The scoring method is theirs.
- 'Accuracy' hides whether sources were checked.
Build your own test set
- Pick 6–10 real tasks.
- Write the known-good answer.
- Add traps (an authority that does not exist).
- Score accuracy, citations and honesty about uncertainty.
What to score
Not 'did it sound good' but 'was every claim traceable'.
Written for legal professionals. Not legal advice. Always verify every authority and let a named lawyer sign off.