🔍 Read the full analysis: About 1 In 5 Tasks Pass For Z.ai’s Legal AI—Yet Its Rank Stays High on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh of 81 models in the archived ranking. The score counts a task as passed only when every rubric criterion is met, while Vals reported a 92.5% pass rate across individual criteria. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the leaderboard figures, but its “state-of-the-art among open-source models” claim lacks a stated comparison set.
Z.ai’s GLM-5.3-FlashX-Legal passed every required criterion on 19.17% of tasks in Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models. The score reflects an all-or-nothing grading rule: a task fails if even one required criterion is missed, so the result does not mean the model got only one-fifth of its individual legal judgments right.
Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026, lists GLM Legal at 19.17% and an evaluation cost of $0.15 per test. Z.ai’s documentation rounds the score to 19.2% and says the model improved by 12.5 percentage points over the base GLM-5.3-Flash. The leaderboard figures support that comparison: the base model scored 6.67%, making the difference exactly 12.50 percentage points.
The benchmark’s other measure gives a more favorable but different view. Vals reports GLM Legal met 92.5% of individual criteria; in that leaderboard version, a criterion counted as passed only when both AI judges passed it. A high criterion-level score can coexist with a much lower task-level score because each task may contain many requirements, all of which must be satisfied for an all-pass result.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including legal research, contract review and drafting. The company says it received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of modeled legal-workflow environments. Z.ai recommends using it as an agent connected to services for legal materials and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens.
Why Full-Task Passes Matter
The ranking and pass rate answer different questions. Seventh place shows GLM Legal scored relatively well against the models included in this particular evaluation; 19.17% all-pass shows how often it cleared every requirement on a task. For legal teams, both figures matter: a model can perform strongly on many individual checks yet still leave at least one issue that requires a human to catch before a work product is relied on.
That distinction is especially relevant because benchmark tasks represent complete work products, not isolated answers. Harvey has argued that a deal report identifying eight of ten risks may still be materially incomplete. The benchmark’s strict grading captures that kind of failure, but it also means its task pass rate should not be mistaken for a general measure of accuracy, usefulness or readiness for unsupervised practice.
Procurement decisions also involve questions the leaderboard does not resolve. Firms considering the model would need to establish how client information is processed, what rules apply to stored or transmitted data, and who verifies authorities, citations and conclusions. The source material describes model outputs as drafts for legal professionals. The benchmark result alone does not establish that the system can safely replace professional review.
Popular right now
Vitamix 5200 Professional-Grade Blender, 64 oz. Container, Black
KitchenAid Classic Series 4.5-Quart Tilt-Head Stand Mixer, Onyx Black
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public benchmark contains 1,250 tasks spanning 24 legal practice areas, assessed using more than 75,000 expert-written rubric criteria. Tasks provide an instruction and matter materials, then ask an agent to produce work for review. Criteria can cover facts, conclusions, citations, deadlines, recommendations, amounts and formatting.
Harvey’s grading rule is called all-pass grading: every required criterion must pass for the task to count as complete. Vals AI operates a public leaderboard using a private holdout set of 120 tasks, according to the supplied benchmark details. Internet access is disabled during evaluation. Two language-model judges, GPT 5.5 and Claude Sonnet 4.6, grade submissions; the reported task score averages their pass rates.
Z.ai’s AutoClaw blog post announcing the model is dated October 13, 2026, while Vals lists the release as October 12. Z.ai’s documentation does not carry a date. Its statement that the model achieved “state-of-the-art performance among open-source models” is a company claim; the cited documentation does not define the comparison group, and the supplied material says published weights for the legal model could not be found.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its Legal Agent Benchmark grading rule
Limits of the Leaderboard Result
The score is tied to Vals AI’s archived Version 1 leaderboard and its 120-task private holdout, not a guarantee of performance on other legal work. The supplied material also references a later Vals benchmark version updated October 30, 2026, but does not provide enough results to establish how GLM Legal ranks on that version. The leaderboard’s results may depend on its task mix, grading setup and the models included.
A separate four-task spot check described by Thorsten Meyer found citations to rulings that could not be located in outputs from the Z.ai model. That small check is not a benchmark and cannot establish a general error rate. In the same check, OpenAI and Claude models reportedly did not invent rulings, while two Claude models cited a real ruling for the opposite of its holding. These observations are limited and do not replace a controlled comparison.
It is also unclear from the supplied material whether the legal model’s weights are publicly available, what exact data-handling terms apply to client materials, or how its outputs perform across jurisdictions and live legal matters. Z.ai’s documentation recommends connecting the model to external legal and company-data services, but provides no independent evidence here about the reliability of those connections or the verification steps users should apply.
Questions Before Legal Deployment
The next useful evidence would be results from later leaderboard versions, with their task sets, judging methods and model entries clearly identified. Z.ai could also clarify the comparison group for its open-source performance claim and publish details about the legal model’s availability and evaluation methods. The supplied material does not give a date for another benchmark update or a further company announcement.
For prospective users, the immediate next step is not to treat the rank as a deployment decision. Firms will need to test the model on representative work, check citations against primary sources, review data-processing arrangements and define who approves drafts before they are used. Those controls are especially relevant when a task’s benchmark score can fall to a fail because of one missed requirement, even when most individual criteria are met.
Key Questions
What does the 19.17% score mean?
It means the model passed every required criterion on 19.17% of tasks in the Vals AI leaderboard evaluation. A task with even one failed criterion does not count as a pass.
How can the model have a 92.5% criteria pass rate?
The 92.5% figure applies to individual criteria, while 19.17% counts whole tasks where all criteria passed. A task can have many criteria and fail overall if one or more are missed.
What was GLM Legal’s leaderboard rank?
It ranked seventh of 81 models on Vals AI’s archived Version 1 leaderboard, updated October 18, 2026.
Did Z.ai improve on its base model?
Yes, according to the leaderboard figures cited: GLM Legal scored 19.17%, compared with 6.67% for GLM-5.3-Flash. That is a 12.50-percentage-point difference, matching Z.ai’s rounded 12.5-point claim.
Does the score show the model is ready to handle legal work without review?
No. The benchmark measures performance on a specific set of tasks under a defined grading system. It does not establish safe, unsupervised use in live matters; the source material describes outputs as drafts for legal professionals.
Source: ThorstenMeyerAI.com
