The Best AI For My News Pipeline Depends On More Than Test Results

🔍 Read the full analysis: The Best AI For My News Pipeline Depends On More Than Test Results on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI model configurations for his news-writing pipeline found that GPT-6 Astra produced the highest-scoring articles, but he kept cheaper GPT-6 Luna in production and removed two backup models after they generated invented quotes or invalid output. The test, conducted from September 8 to October 13, 2026, measured models on real news sources and the production prompt rather than relying on general-purpose leaderboards.

In the standard test, Astra scored 4.00 out of 5 in both English and German and received a perfect source-fidelity score. In a follow-up run covering 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reports a cost of about $0.12 per article and an average time of roughly 50 seconds.

Luna scored 3.67 in English and 3.68 in German in the standard test. It averaged 3.67 across the robustness run, with source fidelity of 4.71 out of 5, and all its test articles were valid under the pipeline’s output-repair rules. Meyer says it cost about $0.0012 per article in the test and about $0.0013 in production during the previous week, taking a median of 17 seconds. He chose it over Astra on cost grounds.

The test also found that raising Luna’s reasoning setting did not produce a clear quality gain. At the maximum setting, an article cost about 12 times more and took about 12 times longer than at the production setting, while scores remained within the test’s reported noise. Meyer also reports that GPT-4.1 nano scored 1.62 in English and 1.68 in German, with the lowest source-fidelity score, and that locally run gpt-oss-120b produced no valid articles in four attempts. Both were removed from the news route on October 7.

Why Cheap Reliability Won

The results illustrate a practical distinction for publishers: the highest-scoring model is not automatically the best operational choice. Meyer reports Astra performed better, but at roughly 100 times Luna’s per-article cost. For a high-volume pipeline, that price gap can outweigh a modest quality advantage if the less expensive model consistently meets the publisher’s standard.

The test also points to a separate operational risk: fallback models can affect published accuracy when the primary service is unavailable. Meyer says his backup configurations produced fabricated quotations, including attributions to unnamed or apparently invented sources. Those are not merely formatting defects; fabricated quotes can mislead readers and damage trust. His account supports testing the entire failover chain, not just the model normally used.

These findings are specific to one publisher’s prompt, source material, evaluation method and cost assumptions. They do not establish that Luna is the best choice for other newsrooms, or that additional reasoning effort is unhelpful for every task. They do show why rankings from unrelated benchmarks may not answer a newsroom’s own production question.

How the Newsroom Test Worked

Meyer describes a test designed to resemble his publishing workflow: models received the production prompt and real news sources, then generated articles in English and German. The source set included a 6,340-character product filing and a much shorter, 619-character political news item. He says most of his newsroom’s starting material is short, making the thin-source test particularly relevant to his use case.

Two AI judges—GPT-6.1 Sol, set to high reasoning, and Claude Opus 5.5—rated articles on a five-point scale for source fidelity, fabrication, clarity about evidence, news value, structure and polish, with German-language quality also assessed. A stronger subset of configurations then wrote 10 additional articles from real sources, mostly short ones. Meyer says he treated score gaps below about 0.3 points in the two-article standard test as potentially noisy and relied more on the robustness run for rankings.

The test counted an article only if it produced valid output after the same repair step used in production. That makes format reliability part of the reported result, but the assessment remains an evaluation published by Meyer on his own pipeline; the material provided does not describe an independent replication.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, describing his model-selection rule

Limits of the Reported Scores

The results are Meyer’s own evaluation, not a peer-reviewed study or an independent comparison. The source material does not provide the full set of model configurations, article outputs, judge-level scores or detailed uncertainty intervals for every result. It also does not establish whether other publishers using different prompts, languages, source types or editorial review would see the same ranking.

The judges were AI systems, and one of them—Claude Opus 5.5—was also among the models being evaluated. Meyer reports using two judges and a separate robustness run, but the material does not say whether human editors independently checked the scores or how disagreements between judges were handled. The reported numbers should be read as evidence about this pipeline, not as a universal ranking of writing models.

The source says GPT-6 Luna now runs first, but its description of the current failover chain ends with “GPT-6 Luna as its” and is incomplete. It is therefore unclear what model, if any, now serves as the backup, and what checks prevent another fallback from returning invalid or fabricated material.

What the Pipeline Must Test Next

Meyer says Luna remains in production at the reported low per-article cost and median 17-second response time, while Astra was rejected for routine news because of its price. The immediate operational priority is to confirm that the updated route has a dependable backup: the published account does not identify the complete failover setup after the October 7 removals.

For readers and publishers following the results, the next useful evidence would be more articles tested across a broader range of stories, plus a clear account of human review, fallback behavior and any changes to the prompt. Meyer’s own method points toward repeat testing as models and costs change. Until those details are available, the finding is best understood as a report about one newsroom’s measured trade-off: higher peak quality from Astra, lower-cost production from Luna, and a need to scrutinize backups.

Key Questions

Which model scored highest in the reported test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test, with perfect source fidelity. It averaged 3.93 in the 12-article robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored consistently well and every test article it wrote was valid, while costing about $0.0012–$0.0013 per article. He reports Astra cost about $0.12 per article, roughly 100 times as much.

Did higher reasoning effort improve Luna’s articles?

Not clearly in this test. Meyer reports that score differences across six settings were within the noise, while the maximum setting cost and took about 12 times more than the production setting.

Why were two fallback models removed?

Meyer says GPT-4.1 nano produced invented quotes and had the lowest source-fidelity score. Locally run gpt-oss-120b produced no valid articles in four attempts and also generated invented named quotes.

Does this prove Luna is the best news-writing model?

No. The results apply to Meyer’s prompt, sources, languages and evaluation process. They do not establish that other publishers will get the same ranking or that the test has been independently replicated.

Source: ThorstenMeyerAI.com