MNanoBEIR / NanoBEIR-en / NanoFEVER

Overview#

NanoFEVER is the compact English NanoBEIR version of FEVER, a fact verification evidence retrieval task. Each query is a short declarative claim, and the corpus contains Wikipedia-style passages that can support or refute the claim. The retrieval goal is to surface evidence-bearing documents for a downstream verifier, not to answer a question directly. This makes the task useful for evaluating claim-to-evidence retrieval, entity grounding, relation matching, and retrieval for false or partially false claims.

Details#

What the Original Data Measures#

FEVER was created for fact extraction and verification over Wikipedia. The original benchmark requires a system to retrieve evidence and then decide whether a claim is supported, refuted, or not supported by available evidence. In the retrieval setting, the important question is whether the evidence page or passage is ranked high enough to be used.

This differs from ordinary question answering because the query is a claim, and the claim itself may be false. A retriever must not assume the wording is correct. It must retrieve the page that contains the decisive evidence about the entity, date, occupation, nationality, title, location, or relation.

Observed Data Profile#

The task contains 50 queries, 4,996 documents, and 57 relevance judgments. Most queries have one positive evidence document, with an average of 1.14 positives per query. The minimum is 1, the median is 1.0, the maximum is 3, and 6 queries are multi-positive, or 12.0% of the set.

Queries average 45.42 characters, while documents average 1,228.71 characters. The short claim must be matched to a much longer evidence passage. Many claims include entity names or titles, but the relevant evidence may be determined by a specific relation or correction rather than broad topical overlap.

BM25 Evaluation Profile#

BM25 reaches nDCG@10 of 0.8143, hit@10 of 0.9400, and recall@100 of 1.0000 using the top-500 BM25 candidate subset. This is a very strong lexical profile. FEVER claims often contain rare entity names, media titles, or factual anchors that appear in the relevant Wikipedia passage.

The remaining difficulty is top-rank evidence selection. BM25 may retrieve related pages about the same person, work, place, or year before the passage that actually verifies the claim. Lexical overlap is excellent for candidate generation, but it can confuse same-entity context with decisive evidence.

Dense Evaluation Profile#

The dense harrier-oss-270m run reaches nDCG@10 of 0.8816, hit@10 of 0.9800, and recall@100 of 0.9825. Dense retrieval is the strongest direct top-rank profile. It improves nDCG@10 and hit@10 over BM25, showing that embedding similarity helps connect claims to evidence relations beyond exact term overlap.

This is particularly useful for refuted or mutated claims. The relevant passage may contradict the claim through a profession, date, nationality, genre, or title relation. Dense retrieval is better at ranking evidence-bearing passages above adjacent same-topic pages, although BM25 retains slightly better complete recall@100.

Reranking Hybrid Evaluation Profile#

The reranking_hybrid candidate set reaches nDCG@10 of 0.8521, hit@10 of 0.9800, and recall@100 of 1.0000. It uses exactly 100 candidates per query, with no safeguard rows. The hybrid profile matches BM25's complete recall@100 and dense retrieval's hit@10, while sitting between them on nDCG@10.

This makes reranking_hybrid a strong evidence candidate pool. BM25 contributes exact entity and title coverage, while dense retrieval contributes relation-sensitive ranking. A downstream verifier or reranker should benefit from the complete candidate availability while still seeing dense-like top-10 visibility.

Metric Interpretation for Model Researchers#

Because most queries have one positive, hit@10 measures whether the evidence is visible, and nDCG@10 measures how early it appears. recall@100 is especially important for multi-stage fact verification because a verifier cannot recover if the evidence page is absent from the candidate pool.

The comparison shows that BM25 is excellent for recall, dense retrieval is best for direct evidence ranking, and reranking_hybrid offers the best reranking-ready tradeoff. This task is useful for evaluating whether a model retrieves evidence, not merely same-topic Wikipedia text.

Query and Relevance Type Tendencies#

Queries include claims such as Keith Godchaux knew the Grateful Dead, Taarak Mehta Ka Ooltah Chashmah is a sitcom, advanced airplanes were produced in Burbank, Nero is a person, and Scream 2 is exclusively a German film. Relevant passages usually contain the entity description or factual relation needed to support or refute the claim.

The task rewards precise entity grounding and claim relation matching. A passage can mention the right entity yet fail to settle the claim. Evidence for false claims is especially challenging because the correct passage often contradicts rather than repeats the query.

Representative Failure Modes#

Likely failures include retrieving pages that share the same entity but lack the decisive evidence, ranking generic pages above the specific evidence passage, confusing similar titles, and treating false claim wording as if it were correct. BM25 may over-reward surface overlap, while dense retrieval may occasionally miss rare exact entity anchors.

Training Data That May Help#

Useful training data includes non-overlapping FEVER training claims with evidence, claim-verification datasets with evidence annotations, Wikipedia claim-evidence pairs, and synthetic claim mutations based on entity, date, relation, and profession substitutions. Upstream FEVER or BEIR-derived evaluation examples should be excluded when overlap is possible.

Model Improvement Notes#

A model targeting this task should preserve high entity recall while improving evidence relation ranking. Sparse systems need strong title and alias handling. Dense systems should train on claim-to-evidence pairs rather than only question answering. Hybrid systems are promising because the observed profile keeps complete recall while improving top-10 visibility.

Example Data#

Source Reference Table#

Representative query and positive evidence snippets:

Dataset Information#

Candidate Subsets#

Training and Leakage Metadata#

• Original train split: available
• Evaluation split origin: unknown
• Train/eval overlap audit: not_audited
• Leakage note: prefer excluding upstream dev/test data or other FEVER/BEIR-derived records likely to overlap with the NanoBEIR evaluation claims and evidence
• Multi-positive training: useful_but_limited
• Useful training data: non-overlapping FEVER training claims with evidence, claim-verification datasets with evidence annotations, Wikipedia claim-evidence retrieval pairs