TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s leaderboard. The score counts a task as passed only when every rubric criterion is met; the same run passed 92.5% of individual criteria. Z.ai’s claimed open-source lead and the model’s reliability on real legal work remain less clear.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh among 81 models on Vals AI’s Harvey Legal Agent Benchmark leaderboard, with a 19.17% all-criteria pass rate. The result, independently evaluated by Vals according to Z.ai, means the model met every required criterion on about one in five tasks; it does not mean it achieved that accuracy on every legal question or workflow.
The score uses all-pass grading: a task counts as complete only if the submitted work product satisfies every criterion in its rubric. A missed citation, deadline, or other required detail can fail the task even when much of the rest is correct. On the same run, the model met 92.5% of individual criteria, a substantially higher figure that measures a different unit than the task pass rate.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as a model for legal document understanding, reasoning, and task execution, including research, contract review, and drafting initial work products. The company says it used legal-domain mid-training and post-training, with reinforcement learning across tens of thousands of simulated legal-workflow environments. The model is built on GLM-5.3-Flash, and Z.ai recommends using it as an agent connected to services for laws, cases, and company data.
Vals AI’s archived leaderboard lists the model at 19.17% and a cost of $0.15 per test. Z.ai’s pricing page separately lists API rates of $0.37 per million input tokens and $1.25 per million output tokens. Z.ai says the benchmark score is a 12.5 percentage-point improvement over the base model; the leaderboard comparison supports that difference. Its further claim of state-of-the-art performance among open-source models depends on the comparison set, which Z.ai does not name in the cited statement.
What Z.ai’s Legal AI Results Say About Its Ranking
GLM-5.3-FlashX-Legal ranked seventh on a legal benchmark, but its task pass rate and open-source claims need careful interpretation.
Two scores, different measures
A task passes only when every rubric criterion is met; criterion-level results count each item separately.
- Measures complete benchmark tasks.
- One missed requirement can fail the whole task.
- Measures individual rubric items.
- Many items can pass while a task still fails.
What the leaderboard establishes
The result compares submitted systems within this benchmark’s setup.
Benchmark performance
Vals AI lists a 19.17% task pass rate and seventh place among 81 models.
Open-source lead
Z.ai claims state-of-the-art performance among open-source models, but the cited comparison set is unspecified.
Questions before legal use
The benchmark does not establish reliability across live matters or firm workflows.
- Verify citations and legal authorities in every draft.
- Assess client-data handling and access controls.
- Keep human review and approval in the workflow.
What the Ranking Shows
The ranking places the legal model near the top of this particular leaderboard, while the 19.17% score shows how demanding the benchmark’s all-or-nothing task measure is. These facts can coexist: a model can rank seventh relative to other submitted systems and still fail at least one required criterion on most tasks. The rank is a comparison within the listed 81 models, not a guarantee of readiness for legal practice.
For law firms and legal teams, the gap between the 92.5% criterion pass rate and 19.17% task pass rate highlights why aggregate scores need careful reading. A document can satisfy many rubric items yet still omit a detail that makes the full work product unsuitable. The benchmark offers a structured comparison, but procurement decisions also turn on whether outputs can be checked, how client information is handled, and how the tool fits existing review procedures.
A small spot check described in the source material found that the legal model cited rulings that could not be located in four tested tasks. That is an informal observation, not a benchmark result, and four tasks cannot establish a general error rate. It does illustrate why citations and legal authorities need verification before a draft is relied on.
Products Worth Considering
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark (LAB) as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction and a client matter folder with a requested work product. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts, and formatting.
Harvey says a task passes only when every rubric criterion passes. The benchmark’s leaderboards use a private holdout set designed to mirror the public task mix. Vals AI says that holdout contains 120 tasks, disables internet access, and uses two language-model judges—GPT-5.5 and Claude Sonnet 4.6—to grade each submission. The final score averages the judges’ task pass rates.
Z.ai’s AutoClaw blog announcement, which opened a beta by application, is dated October 13, 2026; Vals lists the model release as October 12. The developer documentation carrying the benchmark claim has no date. These dates describe the release and announcement, while the score itself is presented on Vals AI’s leaderboard.
““an all-criteria pass rate of 19.2%””
— Z.ai developer documentation
Limits of the Published Result
The leaderboard establishes how the model scored under this benchmark’s setup, but it does not show how it performs across every legal matter, jurisdiction, or firm workflow. The source material does not provide a broader deployment record or evidence that benchmark performance predicts outcomes in live client work.
Z.ai calls the result state of the art among open-source models, but its cited statement does not identify the models included in that comparison. The source material also says it found no published weights for the legal model, leaving the practical basis for describing it as open source unclear. Z.ai’s recommendation to connect the model to legal and company-data services also leaves procurement questions about data processing and access controls unresolved in the cited material.
The four-task spot check is too small to quantify citation reliability. The source material reports that some other tested systems also made legal-citation errors, including citing a real ruling for the opposite of its holding. Those observations are not benchmark findings and do not establish comparative error rates.
Questions Before Legal Use
The next useful evidence would include clearer disclosure of the open-source comparison set, availability of model weights, and further evaluations across legal tasks beyond the current leaderboard. The cited materials do not announce a date for such releases or evaluations.
For organizations considering the model, practical review should focus on citation checking, client-data handling, and human approval of work products. Z.ai describes outputs as drafts for legal professionals; the source material does not report an independent assessment of how firms have deployed the model or what data protections apply in specific configurations.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks for which the model passed every criterion in the task rubric, as listed on Vals AI’s leaderboard. A task fails if at least one required criterion is missed.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The figures use different units. The 92.5% figure counts individual rubric criteria; the 19.17% figure counts whole tasks, each of which passes only when all its criteria pass.
What was the model’s leaderboard position?
Vals AI’s archived leaderboard lists GLM-5.3-FlashX-Legal seventh among 81 models, with a 19.17% task pass rate and a listed cost of $0.15 per test.
Does the ranking establish that the model is ready for legal practice?
No. It reports performance on a defined benchmark. It does not establish reliability across live legal matters, and Z.ai describes the model’s output as drafts for legal professionals.
Is Z.ai’s open-source claim independently established?
The cited Z.ai statement claims state-of-the-art performance among open-source models, but it does not name the comparison set. The source material also reports that it found no published weights for the legal model, so the claim’s basis remains unclear.
Source: Thorsten Meyer AI
