NanoMTEB-Dutch / fever

Overview

fever is the Dutch FEVER retrieval task from BEIR-NL. Queries are Dutch translations of factual claims, and documents are Dutch-translated Wikipedia evidence passages. The Nano split contains 200 queries, 10,000 documents, and 233 positive qrel rows. Most queries have one positive, but 27 queries have multiple positives, and the maximum number of positives for one query is six. It evaluates whether a retrieval model can find evidence-bearing passages for fact verification.

This task has very high retrieval scores across BM25, dense, and reranking_hybrid candidate sources. BM25 is slightly strongest in nDCG@10, and hybrid search has the highest recall@100 while matching BM25's hit@10. Dense retrieval is also strong but loses some recall. The task is useful for studying evidence retrieval where entity overlap is powerful but not sufficient: a model must retrieve passages that support or refute a claim, not merely pages that mention the same entity.

Details

What the Original Data Measures

FEVER: a Large-scale Dataset for Fact Extraction and Verification introduced FEVER as a fact-verification dataset built from claims generated by altering Wikipedia sentences and verifying them against Wikipedia evidence. Claims are labeled as supported, refuted, or not enough information, and the evidence often comes from one or more Wikipedia sentences. In BEIR, FEVER is used as a retrieval task: the claim is the query and the model retrieves evidence-bearing Wikipedia passages.

BEIR-NL translates public BEIR datasets into Dutch. This split should therefore be read as Dutch-translated FEVER evidence retrieval, not as a natively written Dutch fact-checking corpus. Translation preserves much of the entity structure, while changing the language surface around claims and evidence passages.

Observed Data Profile

The split has 200 claims and 10,000 documents. There are 233 positive qrel rows, with an average of 1.165 positives per query. The median is one positive, but 13.5% of queries have more than one positive. Queries average 54.87 characters, while documents average 445.71 characters and usually contain a Wikipedia page title plus a short explanatory passage.

Representative claims involve the Duke of York, Burbank, Adobe Photoshop, Joseph Merrick, and Vic Mensa. Some claims are supported by the positive passage, while others are contradicted by it. The retrieval task is not to predict the label directly; it is to retrieve the evidence needed for verification.

BM25 Evaluation Profile

BM25 reaches nDCG@10 = 0.9221, hit@10 = 0.9800, and recall@100 = 0.9700 over top-500 candidate lists. This is a very strong lexical baseline. FEVER claims often contain the named entity that identifies the relevant Wikipedia page, and the evidence passage repeats key names, dates, titles, or relations.

The remaining difficulty is evidence specificity. A lexically matching page can still be insufficient if the claim asks for a particular date, role, relation, or negated fact. BM25 can retrieve the right article title but may not always place the most useful evidence passage first when several passages share the same entity.

Dense Evaluation Profile

Dense retrieval with harrier_oss_v1_270m reaches nDCG@10 = 0.9207, hit@10 = 0.9400, and recall@100 = 0.9313. Dense retrieval is strong, but slightly below BM25 and hybrid retrieval on recall and hit rate. This suggests that the dense model captures claim-evidence meaning well, but exact entity and date matching remain critical in this FEVER-style corpus.

Dense retrieval is most valuable when the evidence relation is paraphrased or when the claim's wording differs from the passage. Its likely failure mode is semantic overgeneralization: retrieving a passage about the right entity but not the specific fact needed to verify the claim.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate column reaches nDCG@10 = 0.9215, hit@10 = 0.9800, and recall@100 = 0.9785, with exactly 100 candidates per query and no rank-101 safeguard rows. It has the best recall@100 and matches BM25's hit@10, while BM25 has a tiny nDCG@10 advantage. The hybrid profile is therefore the best candidate source for reranking, especially because some queries have multiple positives.

Hybrid search works well here because sparse and dense signals reinforce each other. BM25 captures exact entity and phrase overlap, while dense retrieval can add semantically related evidence passages. A reranker can then focus on which passage actually supports or refutes the claim.

Metric Interpretation for Model Researchers

Unlike the single-positive tasks, this split has 233 positives for 200 queries. nDCG@10 can reward ranking any relevant evidence passage highly, and recall@100 should be interpreted across all positive qrels, not just a single target per query. Multi-positive training objectives are more appropriate than forcing one evidence passage to represent the entire claim.

The near-tie among BM25, dense, and hybrid in nDCG@10 means that small differences should be interpreted carefully. The broader conclusion is that entity-aware retrieval is already strong, and remaining improvements come from evidence selection and relation-level matching.

Query and Relevance Type Tendencies

Queries are Dutch factual claims. They often contain a named entity plus a predicate such as a birth date, title, occupation, location, release fact, or historical relation. Relevant documents are Wikipedia evidence passages that can support or refute the claim.

Relevance is evidence bearing. A document about the same entity is not always enough; the passage must contain the fact needed to verify the claim. For multi-positive claims, several passages may independently provide enough evidence or contribute related evidence.

Representative Failure Modes

BM25 can fail when the right entity has many candidate passages and the evidence relation is not the most lexically obvious one. Dense retrieval can fail by retrieving a semantically related entity page or same-topic passage that lacks the verifying fact. Hybrid retrieval can still surface entity-near distractors, especially when several passages share names and dates.

Hard negatives should be related Wikipedia passages that mention the same entity but do not support or refute the claim. These are more useful than random negatives because the task is already easy at the entity-matching level.

Training Data That May Help

Useful training data includes official FEVER training claim-evidence pairs with overlap removed, Dutch and multilingual fact-verification retrieval data, non-overlapping Dutch Wikipedia claim-evidence pairs, and hard negatives from related Wikipedia pages. Training should exclude the translated FEVER test queries, qrels, and positive evidence passages used by this Nano split.

Synthetic data can be generated from Dutch encyclopedia passages outside the evaluation set. Create factual claims that are explicitly supported or contradicted by the passage, preserving entities, dates, and relations. Multi- positive examples should be retained when several passages genuinely support the same claim.

Model Improvement Notes

Improving this task requires evidence-aware reranking rather than basic entity retrieval. Models should learn to distinguish passages that merely mention an entity from passages that verify the claim's specific relation. Date, title, occupation, location, and negation cues are especially important.

For rerankers, the hybrid pool is a strong starting point. It provides high candidate coverage, and the remaining challenge is relation-level evidence selection. Multi-positive supervision should be preserved so that the reranker does not penalize alternative valid evidence passages.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: translated BEIR-NL FEVER test split from clips/beir-nl-fever
• Train/eval overlap audit: not_audited
• Leakage note: Do not train on the translated FEVER test queries, qrels, or positive evidence passages used by this Nano split.
• Multi-positive training: multi_positive_objective
• Useful training data: official FEVER training claim-evidence pairs with overlap removed, Dutch and multilingual fact-verification retrieval data, non-overlapping Dutch Wikipedia claim-evidence pairs, hard negatives from related Wikipedia pages