NanoMTEB-Dutch

Overview

NanoMTEB-Dutch is a compact Dutch retrieval group covering translated BEIR-NL tasks, native Dutch MTEB tasks, cross-lingual Belebele retrieval, legal and medical retrieval, scientific evidence retrieval, news and tender retrieval, web FAQ retrieval, and Dutch Wikipedia retrieval. Ten of the twenty-seven tasks are Dutch CQADupStack duplicate-question splits, but the group is broader than duplicate QA.

The group should be read as both a Dutch-language benchmark and a translation robustness benchmark. Some tasks are native Dutch resources, such as legal, news, tender, FAQ, or bibliography retrieval. Others carry BEIR-style relevance relations into Dutch. BM25 exposes exact Dutch terms and named entities; dense retrieval tests paraphrase, translation, and cross-lingual matching; reranking_hybrid is useful when sparse and dense candidates recover different positives.

What This Group Measures

NanoMTEB-Dutch follows Dutch retrieval coverage assembled for MTEB-NL and BEIR-NL. It includes translated BEIR-style datasets, cross-lingual Belebele directions, Dutch legal resources, Dutch news, public procurement, Flemish academic bibliography, FAQ retrieval, and Wikipedia retrieval.

The group measures Dutch retrieval across task semantics. A relevant document may be a duplicate question, a statute article, a FAQ answer, a news article, a procurement notice, a scientific paper, a medical abstract, or a Wikipedia passage. A model must preserve both Dutch lexical details and the original task relation after translation or cross-lingual mapping.

Task Families

• Duplicate and argument retrieval: argu_ana_nl, the ten cqadupstack_* tasks, and quora.
• Cross-lingual and monolingual reading comprehension: the three Belebele directions.
• Legal retrieval: b_bsardnl and legal_qanl.
• Open-domain and encyclopedic retrieval: nq, fever, and wikipedia_multilingual_nl.
• Scientific and medical retrieval: nfcorpus_nl, sci_fact_nl, and scidocs_nl.
• News, procurement, bibliography, and FAQ retrieval: dutch_news_articles, open_tender, vabb, and web_faq_nld.

Dataset Shape

NanoMTEB-Dutch contains 27 task pages, 5,299 queries, 227,987 split-local documents, and 13,018 positive qrel rows. Most tasks have 200 queries. The group has a mix of single-positive and multi-positive tasks; NFCorpus, SCIDOCS, Quora, LegalQA-NL, bBSARD, FEVER, and NQ require multi-positive interpretation.

Text lengths and formats are diverse. Argument queries are long, FAQ and web queries are short, legal tasks contain formal statute language, CQADupStack contains technical forum text, and scientific tasks contain paper or abstract language. This makes task-family breakdown more useful than one aggregate Dutch score.

Retrieval Behavior

BM25 Profile

BM25 is strongest on tasks with direct named entities, titles, article terms, or Dutch surface overlap. FEVER, Dutch news, Wikipedia, Quora, LegalQA-NL, WebFAQ, OpenTender, and VABB all show substantial sparse signal. BM25 is also useful on same-language Belebele where question and passage share Dutch answer context.

BM25 is weaker on cross-lingual Belebele, bBSARD, SCIDOCS, and many CQADupStack technical duplicate tasks. These require language alignment, paraphrase, legal concept mapping, scientific relatedness, or duplicate intent beyond exact word overlap.

Dense Profile

Dense retrieval is the best profile for many Dutch tasks, especially cross-lingual Belebele, Quora, WebFAQ, Wikipedia, VABB, NQ, SciFact, and several CQADupStack splits. It helps when the Dutch query and document express the same intent with different words, or when translated benchmark artifacts weaken literal matching.

Dense retrieval still needs exact anchors. Legal articles, procurement notices, scientific terms, and technical forum posts can depend on specific names, codes, product terms, or statute phrases. Dense gains should be checked against candidate recall for those anchors.

Reranking Hybrid Profile

reranking_hybrid is strongest on same-language Belebele, LegalQA-NL, and several CQADupStack tasks. It is most useful where exact Dutch terms and dense paraphrase matching are both needed. For FAQ, Quora, and Wikipedia-style tasks, hybrid remains competitive even when dense has the best nDCG@10.

For reranker experiments, multi-positive tasks such as NFCorpus, SCIDOCS, Quora, and legal retrieval should be read with recall and candidate diversity in mind, not only first-hit success.

Task Summary

Interpretation Notes for Model Researchers

NanoMTEB-Dutch should be interpreted by source type. Translated BEIR tasks stress whether the original relevance relation survives in Dutch. Native Dutch legal, FAQ, news, tender, and bibliography tasks stress domain vocabulary and local data conventions. Cross-lingual Belebele should be read separately from same-language Dutch retrieval.

The BM25/dense profile is a useful diagnostic. BM25-led tasks show direct Dutch surface anchors. Dense-led tasks show paraphrase, translation, or semantic intent matching. Hybrid-led tasks show that both exact terms and semantic alignment are needed for candidate generation.

Training and Leakage Notes

Useful training data includes Dutch search logs, legal question-article pairs, FAQ retrieval, translated BEIR-NL training data, duplicate-question pairs, scientific and biomedical retrieval, and cross-lingual QA pairs. Hard negatives should be drawn from same legal article families, same technical forums, same scientific topic, or same answer-bearing article.

Exclude NanoMTEB-Dutch evaluation queries, positives, qrels, duplicate clusters, translated test examples, statute articles, and source rows. Cross-lingual tasks should avoid direct translations of evaluation queries as synthetic training seeds.

Source Reference Table

Metadata Summary

Task Metadata Summary