TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A paper from Multiverse Computing introduces ProvenanceGuard, a post-generation verification layer for MCP-based LLM agents that checks whether each claim is supported by the source the answer attributes it to, not just whether it appears somewhere in the pooled evidence. On 361 expert-checked claims from medical agent traces, it caught 138 of 139 claims experts said should fail. Source-routing accuracy was about 86%.
Researchers at Multiverse Computing have published ProvenanceGuard, a verification layer for LLM agents that use the Model Context Protocol (MCP), designed to catch a failure mode the authors call cross-source conflation: claims that are factually true somewhere in the evidence but attributed to the wrong source. In tests on 361 human-expert-checked claims from a medical agent, the system caught 138 of 139 claims the experts judged should not pass, according to the company’s paper and accompanying Hugging Face blog post.
Most existing factuality checkers, including RAGAS faithfulness, MiniCheck, AlignScore, and SummaC, evaluate whether a claim is supported by pooled evidence once multiple sources have been merged, the authors write. That approach breaks down for MCP agents, which may simultaneously query search tools, structured records, and databases. The blog post gives the example of a support agent stating “According to the account record, this plan includes a 30-day refund window” — the refund window may be real, but stated in a policy document rather than the account record. A source-blind verifier passes it; a source-aware one should not.
ProvenanceGuard operates as a post-generation layer over a black-box agent, requiring no retraining. It reads the captured MCP trace, including tool outputs and source IDs, then runs a five-step pipeline: decompose the answer into claims, route each claim to the most relevant source, score whether that source supports it, compare it against the source the answer names or implies, and emit per-claim verdicts plus an answer-level allow-or-block decision. Blocked answers can go through RARR-style repair and be re-verified.
In the evaluated setup, the authors used local models — MiniLM for source routing, a DeBERTa NLI model for support checking, and a local LLM for claim decomposition — so traces could be processed offline. The authors state these models are the configuration they tested, not a requirement, and that hosted alternatives would need separate testing and calibration. The verifier also checks literal values: a number, date, or identifier absent from the source cannot pass merely because a sentence sounds plausible.
Why Attribution Errors Rival False Facts
The paper’s core argument is that in data-sensitive settings, a wrong attribution can be as damaging as a wrong fact. The authors cite a clinical example: a patient-specific medication detail pulled from a patient-history tool becomes misleading the moment an answer presents it as a finding from the medical literature. For regulated domains like healthcare and customer data, the distinction between “supported somewhere” and “supported by the right source” affects whether agent outputs can be audited and trusted.
The system’s conservative decision policy — it held 67 claims the experts considered supported, sending them for review or repair — reflects a design trade-off: favoring a second look at some correct claims over letting unsupported ones through. The authors note this suits settings where getting the source right matters more than speed. The approach is not limited to medicine; the authors say it can be adapted to any agent that records tool outputs and source IDs.
The Shift From Single-Passage to Multi-Tool Agents
: “Factuality checking for LLMs developed largely around retrieval-augmented generation, where an answer draws on a single retrieved passage or a pooled context. MCP changed that baseline: an agent can now call a search tool, inspect a structured patient or account record, query a database, and pull metadata, then weave all of it into one answer. Existing faithfulness scores remain useful, the authors write, but they do not indicate which MCP tool output supports each claim, or whether that matches the source the answer names — the gap ProvenanceGuard targets. The evaluation drew on answers from a medical agent that had used patient records, research articles, and other tools, yielding 281 real traces.
“The failure mode we care about is one we call cross-source conflation: a claim that is true somewhere in the evidence, but attributed to the wrong source.”
— Multiverse Computing, Hugging Face blog post
Limits of the 361-Claim Evaluation
Several limits should be kept in view. The headline result comes from 361 claims across 40 answers held out from the system’s development data — a small evaluation set from a single domain (medicine). The system missed one claim the experts judged should fail, and it over-blocked 67 supported claims. Source routing picked the right source about 86% of the time in this test, meaning roughly one in seven routings went to the wrong source. Results were produced with a specific local-model configuration; the authors state that swapping in hosted models would require new testing and calibration. The blog post text is truncated before fully reporting the comparison against the four other support checkers, beyond stating ProvenanceGuard scored highest on the paper’s measure — readers should consult the paper for complete figures.
Adapting the Method Beyond Medicine
The paper is available on Hugging Face and, per the authors, on arXiv. The authors indicate the same claim, source, and decision steps can be adapted to hosted models and to other domains wherever agents retain records of tool outputs and source IDs. Independent replication on larger and non-medical evaluation sets, and testing with cloud-based model configurations, would be the natural next steps; no timeline for such work is given in the source material.
Key Questions
What is cross-source conflation?
It is a failure mode where a claim is true somewhere in the pooled evidence but attributed to the wrong source — for example, a refund policy fact presented as coming from an account record. Source-blind verifiers pass such claims because the fact exists in the pool.
Does ProvenanceGuard require retraining the agent?
No. According to the authors, it is a post-generation layer that runs on top of a black-box MCP agent, reading the captured MCP trace without modifying the agent itself.
How accurate was it in testing?
On 361 expert-checked claims from 40 answers, it caught 138 of 139 claims experts said should fail, missed one, and held 67 supported claims for review or repair. It picked the correct source about 86% of the time for claims with an identifiable source.
Which models does it use?
The evaluated setup used local models: MiniLM for finding the relevant source, a DeBERTa NLI verifier for support checking, and a local LLM for claim decomposition. The authors say this is the tested configuration, not a requirement, and hosted alternatives would need their own testing.
Can it be used outside medicine?
The authors state the method can be used in other fields whenever an agent keeps a record of its tool outputs and source IDs. Medicine served as the test domain because patient-record facts and general research facts cannot be treated as the same source.
Source: rss
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
