Evaluation

RAGTruth

Can multiple AI agents detect when RAG systems hallucinate?

How it works
  1. Extract claims
  2. Verify
  3. Decide

The problem

Retrieval-augmented systems still produce claims their sources don’t support. Detecting those hallucinations usually needs a trained classifier and labelled data.

How it works

A three-agent pipeline decomposes each response into factual claims, verifies every claim against the source passages, and flags unsupported spans using task-specific decision rules. It was evaluated on RAGTruth across QA, summarization and data-to-text.

The full write-up is on the research page.

03 /

Key decisions

  • Zero-shot on purpose

    No fine-tuning and no GPU, so the pipeline can be dropped into any RAG system without labelled data.

  • Rules decide, agents verify

    Agents extract and check claims; task-specific rules aggregate their verdicts into the final decision.

04 /

Results

F1
0.642
Records
600
Training examples
0
Per record
~$0.0003