📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project built a large-scale European sovereign LLM from scratch, achieving impressive benchmarks but scoring poorly on Italian school exams. This raises questions about the scale needed for country-specific language models. The project exemplifies a different approach but highlights significant challenges.
Italy’s Minerva project, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, revealing a significant gap between technical achievement and real-world language understanding.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, built models ranging from 350 million to 7 billion parameters. Despite outperforming comparable multilingual models on Italian benchmarks, the 3B version’s low score on the INVALSI test—an essential academic assessment—indicates that large-scale training alone may not produce the depth of country-specific knowledge necessary for complex language tasks.
Minerva’s training involved extensive use of Italian data, with 1.14 trillion tokens of Italian content, making it one of the largest Italian-language models publicly available. The project’s open approach, publishing models, data, and code, contrasts with other European efforts like Portugal’s AMÁLIA, which layered specialization onto multilingual models. However, the empirical results challenge the assumption that scale directly correlates with language understanding, especially in complex domains like education.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Advanced Language Tool Kit: Teaching the Structure of the English Language
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Photo Editing for Beginners: Your Road from Novice to Skilled Professional
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The results from Minerva suggest that simply increasing scale and native-language data may not suffice to produce models with deep country-specific knowledge. This finding has critical implications for Europe’s AI sovereignty efforts, indicating that more targeted or larger investments might be necessary to achieve meaningful language and domain expertise. The project exemplifies both the potential and limitations of current large-scale language modeling approaches in a European context, emphasizing the need for re-evaluating strategies around model size and data composition.
European Sovereign LLM Development and Challenges
Italy’s Minerva project represents one of Europe’s most ambitious efforts to develop a sovereign language model, utilizing extensive computational resources and open data. Unlike Portugal’s AMÁLIA, which focused on incremental adaptation of multilingual models, Minerva was trained from scratch on a vast Italian dataset, reflecting a different strategic choice. Despite technical successes, the low performance on complex academic tests reveals ongoing challenges in translating large-scale training into practical language understanding, a concern shared across European projects.
“Minerva’s results challenge the assumption that native-language investment at current scales is sufficient for complex language tasks.”
— Thorsten Meyer, AI researcher
Unresolved Questions About Scale and Language Depth
It remains unclear what scale or specific training approaches are necessary to produce models capable of passing complex national assessments like INVALSI. Further research is needed to determine whether larger models, different training data compositions, or additional fine-tuning can bridge the observed gap in country-specific knowledge.
Next Steps for European Sovereign LLM Projects
The Minerva team plans ongoing iterations, including continued training and evaluation, to better understand the relationship between scale, data, and language understanding. European projects may need to reconsider investment levels and training strategies, potentially focusing on domain-specific or hybrid approaches, to achieve deeper country knowledge. Further benchmarking on real-world tasks will clarify what scale is truly required.
Key Questions
Why did Minerva score so poorly on the Italian INVALSI exam?
The evaluation suggests that despite large-scale training, the model lacks the nuanced understanding required for complex academic assessments, indicating that scale alone may not guarantee deep language comprehension.
How does Minerva’s approach differ from other European models like AMÁLIA?
Minerva was trained from scratch on a large Italian dataset, whereas AMÁLIA layered Italian specialization onto a multilingual foundation through continuation pre-training, reflecting different strategic choices.
What are the implications for Europe’s AI sovereignty efforts?
The findings imply that European projects may need to invest more heavily or adopt different training methods to produce models with sufficient country-specific knowledge, beyond just scaling data and parameters.
Is the low exam score a sign of failure?
No. It highlights the complexity of translating large-scale training into practical language understanding, and underscores the need for further research rather than a definitive failure.
What will the Minerva team do next?
The team plans to continue refining their models, experimenting with different training scales and techniques, to better understand how to achieve deeper country-specific language comprehension.
Source: ThorstenMeyerAI.com
