Scale AI and IMDA Launch SG-LegalClarify Benchmark: 19 Models, 1,860 Scenarios

Scale AI and Singapore's Infocomm Media Development Authority (IMDA) launched SG-LegalClarify on November 12, 2026, a benchmark that tested 19 language models on 1,860 scenarios spanning 10 domains of Singapore law. The benchmark scores two things at once: legal accuracy and whether a model asks for missing facts before answering.

Scale describes SG-LegalClarify as the first research initiative under a collaboration formalized in April 2026 and publicly detailed on July 7, 2026. The project targets a specific gap: governments currently assess AI systems against international benchmarks and assume the results transfer. Scale argues they should test models against their own laws and institutions instead.

Singapore makes a demanding test case, according to the research paper. Inherited English common law operates alongside local modifications and Singapore-specific statutes, so a model leaning on generic common-law reasoning can miss the governing local rule. The stakes are concrete: a driver asking about a speeding penalty may need to supply the vehicle type, actual speed and existing demerit points. An assistant that quotes a penalty anyway can sound authoritative and still be legally wrong.

The takeaway: in legal AI, knowing what you don't know is part of the accuracy problem.

How the benchmark was built

The dataset starts with 1,100 English-language source questions: 340 fully specified and 760 underspecified. Every underspecified question has a fully specified twin containing its hidden facts. Twenty-three verified lawyers wrote the questions, seven of them doubling as reviewers. Their legal experience ran from two to 37 years, median six.

Each question carries a lawyer-written rubric covering issue identification, rule recall, application and conclusion — 12,546 criteria across the dataset. Of those, 89.0% are mandatory for a substantively correct answer, and headline results count only mandatory items.

A sufficiency gate forces each model to choose between answering directly and gathering facts. On the fact-gathering path, a simulated user reveals hidden facts only when the model explicitly asks. A three-model AI judge panel, selected through comparisons with human lawyer assessments, grades each criterion by majority vote.

The takeaway: 23 lawyers and 12,546 grading criteria back every headline number.

What did the results show?

Claude Fable 5 led closed-book testing at 69.8% of mandatory criteria satisfied, against an average of 53.8%. With web search, Kimi K3 led at 77.8% and the average rose to 63.0%. Every model improved, though the gains varied. Rule recall was the weakest component: 43.2% pass rates without search, 56.5% with it.

Clarification behavior stayed weak on both paths. Scale's summary reports that roughly 39% of closed-book scenarios with missing facts received no clarifying question, while models asked unnecessary questions in about 28% of scenarios that already had enough information. When facts were missing, models asked about only 39% of the relevant facts legal experts had identified.

Even perfect information gathering didn't guarantee correctness. In a descriptive comparison of web-enabled trials where models requested every relevant missing fact, 55.6% reached the correct legal conclusion, 32.1% reached a wrong one and 12.3% gave no conclusion at all.

The takeaway: search tools lifted scores, but models still fumbled both when and what to ask.

What comes next?

The authors flag several limits: human validation of the AI helpers was limited, provider differences prevented identical search-tool access, and upfront routing may have altered clarification behavior. Not every annotated missing variable is necessary to resolve an instance, so incomplete fact-request coverage doesn't automatically mean inadequate clarification.

The researchers plan to release 50% of the source questions in each category and retain the rest for private evaluation, keeping every underspecified–fully specified pair in the same split. Model outputs on the public split and the full evaluation pipeline are also slated for release.

The paper's ethics statement is blunt: the scores should not be read as evidence that a model is safe for unsupervised legal advice. For governments weighing legal AI, the findings reframe clarification as part of accuracy itself — a system must recognize which facts local law makes material, then apply that law correctly once it has them.