🔍 Read the full analysis: We Compared 42 AI Setups On The Same Legal Questions on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer AI says its Lexassay test gave 42 AI setups the same 15 legal questions, covering German, US and EU law. Its results ranged from 8 out of 100 for a legal-specific model to 79 for Claude Opus 5.5, but the author says the test is a spot check, not a strict benchmark, and discloses that Claude agents graded the answers.
Thorsten Meyer AI published results from Lexassay, a test that put 42 AI setups through the same 15 legal questions, and reported substantial differences in scores and citation accuracy. The publisher says the legal-specific GLM-5.3 FlashX Legal scored 8 out of 100, while Claude Opus 5.5 led with 79; the publisher also disclosed that Claude agents graded the answers, a potential conflict of interest.
Lexassay’s questions cover German, US and EU law, including legal deadlines, recent changes in law, major court decisions and questions based on false premises. Ten questions and their answers are public; five are hidden, with only their scores released. The models answered without internet access or tools, according to the test description. Different reasoning-effort settings count as separate setups, so the 42 entries do not necessarily represent 42 distinct underlying models.
In the published results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer says the leading model fully met every criterion on only eight of the ten public tasks. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8 and received zero on the hidden set. The maker Z.ai describes its legal model as optimized for professional legal workflows; these scores are results from Lexassay, not an independent finding about performance in day-to-day legal practice.
The publisher reports that citation checks across published runs found 2,644 citations that were real and 804 that were invented or did not support the claims attached to them. Lexassay deducts points for invented or misattributed citations and for answers that omit a legal-advice disclaimer or suggestion to consult a lawyer. Meyer says one answer gave the Berlin appeal deadline a day too late, while another placed the deadline for appeal grounds a month too late. The source warns that a late civil appeal in Germany can be rejected as inadmissible.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Deadlines and Citation Errors
The test highlights a practical risk for people using general chatbots to understand legal questions: an answer can sound specific while getting a deadline, legal authority or its meaning wrong. In the German example described by Meyer, a holiday affected the appeal deadline in Munich but not Berlin. That difference could matter to a litigant, although the example is one test task, not evidence that every model or real-world answer will make the same mistake.
Its citation findings also matter because users may treat references to statutes and court rulings as proof that an answer is reliable. Lexassay’s reported tally suggests that some citations in its tested answers were fabricated or did not support the stated proposition. The results are a reason to verify cited authorities against official sources and not rely on a chatbot for a filing deadline. They do not establish how often such errors occur across all legal AI use.
The comparison is relevant to product claims as well as consumers. A model marketed for legal workflows scored below the leaders in this test, while a separate benchmark cited by Meyer produced a markedly different ordering. That contrast cautions against treating any single ranking as a definitive measure of legal capability.
Popular right now
Philips 2200 Series Fully Automatic Espresso Machine with Classic Milk Frother, 2 Coffee Varieties, Intuitive Touch Display, Ceramic Grinder
Roborock Robot Vacuum and Mop Combo with Self-Empty Dock
eufy C28 Robot Vacuum Cleaner and Mop for Pet Hair, 15,000 Pa
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Answers
Lexassay evaluates each answer against a prepared gold answer containing three to six weighted criteria, according to its method description. It also checks citations against official databases and classifies them as real, misattributed, invented or not findable. A citation may refer to a genuine law or ruling but still be misattributed if the source does not support the model’s claim. Each misattributed or invented citation costs a quarter of the task’s points under the published rules.
Meyer describes the exercise as a spot check, not a benchmark in the strict sense: each model-task combination receives one answer, so small score differences may be noise. The five hidden questions are intended to make it harder for future models to reproduce answers from the public material. Because the models had no internet access or tools, the exercise measures responses from the model alone rather than performance in a workflow with legal research or document retrieval.
The comparison’s independence also has a disclosed limitation. Thorsten Meyer AI runs Lexassay, and several Claude agents graded the answers; Claude Opus 5.5 ranked first. Meyer says the method page and public answers, gold answers and citation checks are available for scrutiny. Those disclosures provide readers with information to assess the test, but they do not remove the potential conflict or make the results an independently conducted evaluation.
“It is a spot check, not a benchmark in the strict sense.”
— Thorsten Meyer AI, describing Lexassay
Limits of the Published Scores
The source does not provide enough information here to independently reproduce the full 42-setup comparison or determine how representative the questions are of actual legal work. With one answer per model and task, score differences may reflect variation as well as underlying capability. The grading system, question selection and weighting are set by Lexassay, whose operator disclosed an interest in the project.
It is also unclear from the supplied material how the reported citation total is distributed among individual models and tasks, or how models would perform when allowed to search current law, use legal databases or work with documents. The source cites a separate Vals AI legal-agent benchmark that ranks some of the same models differently, but the two tests use different setups. Their rankings should not be treated as directly comparable measures.
The results do not show that a particular model is safe or unsafe for every legal task, nor do they establish the rate of error in routine use. They describe answers to a defined set of questions under Lexassay’s rules. The publisher itself advises readers not to rely on chatbot answers for deadlines or matters involving money or an authority.
Checking the Method and Results
Readers can consult the Lexassay method page and the published answers, gold answers and citation checks, which Meyer says are available for review. The five hidden-task answers are not public; only their scores are reported. Further scrutiny of the rules and underlying public examples may help readers judge how the grading decisions were made.
The source material does not announce a date for another test or an independent review. Until further results are available, the reported scores should be read as a limited comparison under specific testing conditions. For a legal deadline, contract or communication with an authority, Meyer advises asking a lawyer and checking legal citations against their original sources.
Key Questions
What did the Lexassay test compare?
It compared 42 AI setups answering the same 15 questions about German, US and EU law. The models answered without internet access or tools, according to the publisher.
Which setup scored highest?
Thorsten Meyer AI reports that Claude Opus 5.5 scored 79 out of 100, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher also disclosed that Claude agents graded the answers.
How did the legal-specific model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and zero on the hidden questions in this test, according to Meyer. That is a result from this particular test, not a general measure of its performance in all legal work.
Can these results show which AI is best for legal work?
Not on their own. Meyer calls Lexassay a spot check rather than a strict benchmark, and says each setup received one answer per task. A separate benchmark cited in the source ranked some models differently.
Should people use AI answers for legal deadlines?
The publisher advises users to check every cited law and ruling and not take deadlines from a chatbot. For a deadline, contract or matter involving an authority, the source recommends consulting a lawyer.
Source: ThorstenMeyerAI.com
