Corpus Reconciliation

What it is

Corpus reconciliation is resolving the contradictions inside a set of documents before that set is treated as a source of truth. It means finding the places where two documents answer the same question differently, deciding which is current, and recording the decision — rather than leaving both in the index and letting retrieval pick. The name is ours. The nearest term in the research literature is corpus-level inconsistency detection, introduced in 2025, which covers the finding but not the deciding.

Why it matters

Most retrieval systems do not do this before indexing. Those systems index whatever exists, retrieve the closest match and ground against it, which means a contradiction in the corpus becomes a contradiction in the answers, delivered confidently and with a citation. The scale is measurable: a 2025 study found at least 3.3% of English Wikipedia facts contradict another fact, and those inconsistencies carry into the benchmarks built on that corpus — 7.3% of FEVEROUS and 4.0% of AmbigQA examples. The work is unglamorous and only partly automated — agentic systems can now surface candidate contradictions, but the best fully automated detector in that study reached an AUROC of only 75.1%, and which document is current is still decided by a person. Reconciliation is not the only lever: a separate 2025 paper on retrieval-augmented generation with conflicting evidence proposes MADAM-RAG, in which agents debate over retrieved documents across multiple rounds. It reports gains over strong RAG baselines of up to 11.40% on AmbigDocs, which requires presenting all valid answers for ambiguous queries, and up to 15.80% on FaithEval, which requires suppressing misinformation — gains on ambiguity and on misinformation, not on corpus-level contradiction. On that paper’s own harder dataset, which mixes ambiguity, misinformation and noise, a substantial gap remains, so the consistency of the corpus still bounds what a downstream system can assert without hedging.