Testing 71 AI Model Configurations Against My Own News Workflow

🔍 Read the full analysis: Testing 71 AI Model Configurations Against My Own News Workflow on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on his news-writing workflow and found that GPT-6 Astra scored highest, while the far cheaper GPT-6 Luna was his choice for production. The results, published October 13 on ThorstenMeyerAI.com, also led him to remove two fallback models after they produced invented quotes or invalid articles.

Meyer ran the test from September 8 to October 13, 2026, using the same prompt as his production pipeline and two real news sources: a 6,340-character product filing and a 619-character political news item. Each configuration generated an English and a German article. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, as well as German language quality.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a perfect source-fidelity score. In a follow-up run of 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. Meyer reports a cost of about $0.12 per article and a generation time of roughly 50 seconds.

GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 out of 5; all its articles were valid under the pipeline’s repair rules. Meyer reports a production cost of about $0.0013 per article and a median generation time of 17 seconds. He says he selected Luna for production because Astra’s higher score came at roughly 100 times the cost.

The Cost of Higher Scores

The findings highlight a practical trade-off for publishers using AI at scale: the highest-scoring model may not be the best fit for a daily workflow. In Meyer’s test, Astra delivered stronger judged writing, but its reported per-article cost was about 100 times Luna’s. For a high-volume operation, that difference can shape whether a quality improvement is worth paying for.

The test also points to a reliability issue beyond the main model. Meyer says two backup systems in his failover chain produced fabricated quotations or articles his pipeline could not use. Those failures matter because a fallback may be invoked precisely when the primary system is unavailable, making its output part of the publishing process even if it is not the usual choice.

These results are a report on one publisher’s workflow, not a general ranking of writing models. They suggest that benchmark performance in coding or broad capability tests does not necessarily predict performance on source-based news writing. Other publishers would need to test models against their own prompts, languages and editorial checks.

How the Workflow Was Tested

Meyer says his test was designed to resemble production rather than a general language-model evaluation. It covered API, subscription and free-tier models, as well as open-weight models running on his hardware. The two source texts differed sharply in length; Meyer says the shorter, 619-character item better reflects much of the short-source material his news workflow handles.

Each article counted only if it produced valid output after the same repair step used in production. Meyer also cautioned that with only two standard-test articles, score differences below about 0.3 points could be noise. He treated a robustness run of 10 additional articles for leading configurations as a more reliable basis for ranking. The reported intervals and scores remain tied to this particular set of sources, prompts, judges and languages.

The test compared reasoning-effort settings for GPT-6 Luna as well. Meyer reports that scores across six settings were broadly similar, while the maximum setting cost about 12 times as much and took about 12 times as long as the production setting. He kept Luna on the lowest, “none,” setting.

“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”

— Thorsten Meyer, in the published test report

Limits of the Model Scores

The report does not establish how the models would perform across a larger or more varied collection of news sources, or in other publishers’ workflows. The standard comparison used two source texts, and although the robustness run added 10 articles for leading configurations, Meyer describes it as a test of his own production needs rather than a broad independent evaluation.

The scores also depend on two AI judges and the criteria they applied. The source does not provide human-editor scoring or an independent replication of the results. It is not clear how the reported rankings would change with different prompts, editorial standards, source types or later model versions. Meyer’s findings about fabricated quotations are attributed to the judges and his validity checks; the report does not establish how often those failures would occur in other settings.

Luna Remains in Production

Meyer says GPT-6 Luna now handles the production route, after the two tested fallback models were removed on October 7. His report does not identify a replacement fallback or provide details of a new failover arrangement. He says Luna will remain at the lowest reasoning-effort setting because higher settings did not show a dependable gain in the test.

The next useful evidence would be results from a larger, continuing set of production articles, including how often outputs require repair and whether source-fidelity scores hold across changing news topics. Meyer has not announced a schedule for another test. For now, the published findings describe a model choice based on his own workflow and the data collected through October 13.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both the English and German standard tests, with a perfect source-fidelity score. It averaged 3.93 across 12 articles in the follow-up run.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna averaged 3.67 in the robustness run and produced valid articles in every test. He reports that it costs about $0.0013 per article, compared with about $0.12 for Astra.

Did higher reasoning effort improve the articles?

Meyer reports no consistent improvement across six GPT-6 Luna reasoning settings. The maximum setting cost about 12 times as much and took about 12 times as long as the production setting, while test-score differences were within the noise he identified.

What problems did the removed fallback models produce?

According to Meyer’s report, GPT-4.1 nano received the lowest source-fidelity score and generated invented quotations flagged by the judges. The locally run gpt-oss-120b produced no valid articles out of four attempts and also generated named quotations Meyer says were invented.

Do these results show which AI model is best for every newsroom?

No. The test reflects Meyer’s prompts, sources, languages and evaluation method. The report does not establish how the rankings would apply to other newsrooms or a broader range of articles.

Source: ThorstenMeyerAI.com