NanoMTEB-Dutch / belebele_eng_latn_nld_latn

Overview

NanoMTEB-Dutch / belebele_eng_latn_nld_latn is the English-to-Dutch Belebele retrieval split: Dutch questions retrieve English passages. The Nano split has 200 queries, 488 documents, and 200 positive qrel rows, with exactly one positive passage per query. Current diagnostics show dense retrieval as by far the strongest top-rank profile, reranking_hybrid as strongest recall@100, and BM25 as much weaker but not useless because named entities and numbers often survive across Dutch and English.

Details

What the Original Data Measures

Belebele is a parallel reading-comprehension benchmark covering 122 language variants over FLORES-200 passages. The retrieval adaptation uses a question as the query and the corresponding answer-bearing passage as the positive document. MTEB-NL includes Belebele retrieval to evaluate Dutch monolingual and cross-lingual retrieval.

This split specifically tests cross-lingual alignment from Dutch questions to English passages. A model must bridge language while preserving the reading-comprehension relation between question and evidence.

Observed Data Profile

The Nano split contains 200 queries, 488 documents, and 200 positive qrel rows. Every query has exactly one positive document. Queries average 69.39 characters, while documents average 475.51 characters.

Queries are Dutch comprehension questions. Documents are English passages. Observed examples ask about a shooting event, arrestee detention rules, the Chandrayaan-1 lunar probe, the Clean Air Act, and the NBA season suspension.

BM25 Evaluation Profile

The dataset-provided BM25 candidate subset contains all 488 documents per query and achieves nDCG@10 = 0.4738, hit@10 = 0.5850, and recall@100 = 0.6150. BM25 has a moderate score for a cross-lingual task because names, dates, acronyms, and cognates can overlap between Dutch questions and English passages.

Still, BM25 is far below dense retrieval. Ordinary Dutch wording such as question forms, negation, causal language, and descriptions has no direct lexical match in the English evidence passage.

Dense Evaluation Profile

The dense harrier_oss_v1_270m candidate subset contains all 488 documents per query and achieves nDCG@10 = 0.8918, hit@10 = 0.9650, and recall@100 = 0.9850. Dense retrieval is the strongest top-rank profile.

This is a clear cross-lingual embedding success case. Dense retrieval can align Dutch question meaning with English passage evidence, including questions about exclusions, reasons, chronology, and entity roles that are not recoverable from surface token overlap alone.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset contains mostly 100 candidates per query, with two queries using a rank-101 safeguard row. It achieves nDCG@10 = 0.6283, hit@10 = 0.7000, and recall@100 = 0.9900. Hybrid retrieval has the best recall@100 but is much weaker than dense retrieval for top-rank quality.

The profile suggests that sparse evidence helps retain positives through named entities and numbers, but it also introduces many lexical false positives. A reranker must rely primarily on cross-lingual semantic evidence.

Metric Interpretation for Model Researchers

This task is single-positive: each Dutch question has one English answer-bearing passage. Hit@10 measures whether that passage appears near the top. nDCG@10 is sensitive to rank, and recall@100 measures whether the passage is available for reranking.

The main signal is cross-lingual reading-comprehension retrieval. BM25 is an entity-anchor baseline; dense retrieval is the meaningful first-stage standard.

Query and Relevance Type Tendencies

Queries are Dutch questions asking about facts, events, reasons, exclusions, or decisions in an English passage. Relevant documents are short English passages from FLORES-style sources.

The task rewards Dutch-English semantic alignment and evidence matching. It penalizes models that only match names without checking whether the passage answers the question.

Representative Failure Modes

BM25 can retrieve English passages sharing a name or date but not answering the Dutch question. Dense retrieval can confuse related passages when several English documents contain similar entities or events. Hybrid retrieval can under-rank the dense-positive passage when lexical anchors point to a wrong entity-neighbor.

Rerankers should compare the Dutch question's requested relation against the English passage, including negation and exclusion questions.

Training Data That May Help

Useful training data includes Dutch-to-English parallel QA retrieval pairs, translated reading-comprehension retrieval examples, multilingual question-passage pairs with Dutch queries and English documents, and sentence-aligned Dutch-English corpora converted to retrieval with overlap removed. Belebele test questions and passages used by this Nano split should be excluded from training.

Synthetic data can use non-evaluation English news or encyclopedic passages and generate Dutch comprehension questions. Hard negatives should share entities or topic words but not answer the question.

Model Improvement Notes

Dense retrievers should preserve Dutch-English alignment for question intent, not only entity names. Sparse systems need translation or expansion to be competitive. Rerankers should perform cross-lingual evidence checking.

For hybrid systems, NanoMTEB-Dutch / belebele_eng_latn_nld_latn is a dense-first task: hybrid retrieval gives excellent recall, but top-rank quality comes from cross-lingual dense similarity.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: unknown
• Evaluation split origin: mteb/belebele eng_Latn-nld_Latn test split
• Train/eval overlap audit: not_audited
• Leakage note: Exclude Belebele test questions and passages used by this Nano split.
• Multi-positive training: single_positive
• Useful training data: Dutch-to-English parallel QA retrieval pairs, translated reading-comprehension retrieval examples, multilingual question-passage pairs with Dutch queries and English documents, sentence-aligned Dutch-English corpora converted to retrieval with overlap removed