MNanoBEIR

Overview

MNanoBEIR is the multilingual NanoBEIR group: a grid of compact BEIR-style retrieval tasks across Arabic, German, Spanish, French, Italian, Japanese, Korean, Norwegian, Portuguese, Serbian, Swedish, Thai, and Vietnamese. Each language variant contains the same thirteen source tasks, so the group separates two questions that are often mixed together: whether a model understands the underlying BEIR retrieval relation, and whether that behavior survives in non-English text.

The source task mix is deliberately heterogeneous. Some tasks retrieve duplicate questions, some retrieve Wikipedia evidence, some retrieve biomedical or scientific documents, some retrieve debate arguments, and some retrieve answer-bearing web passages. A single average score is therefore not enough to understand the group. The useful reading is by language, task family, and retrieval profile. BM25 exposes exact-term and named-entity dependence; dense retrieval exposes semantic transfer and paraphrase handling; reranking_hybrid shows where sparse and dense candidates complement each other.

What This Group Measures

BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models introduced a benchmark philosophy based on many retrieval relations rather than one passage-search task. MNanoBEIR inherits that philosophy from compact NanoBEIR tasks and applies it to multilingual evaluation. The result is a regular 13-by-13 grid: thirteen languages and thirteen BEIR-derived source tasks.

This group measures multilingual robustness under changing relevance semantics. For example, a relevant Quora document should be a duplicate question, a relevant FEVER document should support or refute a claim, a relevant ArguAna document should work as a counterargument, and a relevant SCIDOCS document should be scientifically related. A retriever that treats all rows as generic semantic similarity will miss much of what the group tests.

Task Families

• Argument retrieval: NanoArguAna and NanoTouche2020 evaluate counterargument and debate-passage retrieval. They often have long text and stance-sensitive negatives.
• Evidence retrieval: NanoClimateFEVER, NanoFEVER, and NanoSciFact evaluate claim-to-evidence retrieval. Exact named entities matter, but the target passage must also express the right evidence relation.
• Open-domain QA retrieval: NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, and NanoNQ evaluate answer-bearing retrieval for finance, multi-hop QA, web search, and natural questions.
• Entity and duplicate retrieval: NanoDBPedia and NanoQuoraRetrieval stress entity-page matching and duplicate-question intent matching.
• Scientific and biomedical retrieval: NanoNFCorpus and NanoSCIDOCS are domain-specific and multi-positive, with many lexical traps around medical terms, paper titles, and related-work language.

Dataset Shape

MNanoBEIR contains 169 task pages, 8,437 queries, 737,399 split-local documents, and 61,048 positive qrel rows. Each language has 13 task pages and 649 queries: most base tasks have 50 queries, while NanoTouche2020 has 49. The document count is a sum over task-local pools, not a deduplicated multilingual corpus.

The group is highly multi-positive. NanoArguAna and NanoMSMARCO are single-positive in this grid, but NanoDBPedia, NanoNFCorpus, NanoSCIDOCS, and NanoTouche2020 contain many positives per query. This means hit@10 can look good while nDCG@10 or Recall@100 still shows ranking quality differences. Query and document length also vary by task: ArguAna and Touche have long argumentative text, Quora has short question text, NFCorpus has short medical queries and many positives, and FEVER-like tasks depend on claim-evidence alignment.

Retrieval Behavior

BM25 Profile

BM25 is strongest where exact words, named entities, titles, technical terms, or many acceptable positives dominate. In this group, Quora, FEVER, DBPedia, HotpotQA, and some Touche splits often give sparse retrieval a clear path. BM25 is weaker on finance, scientific related-paper retrieval, and some climate evidence tasks where the relevant document may use different wording from the query.

Language differences should not be reduced to script alone. Latin-script European languages can still differ because of translation choices and morphology; Japanese, Korean, Thai, Arabic, Serbian, and Vietnamese introduce additional tokenization and segmentation considerations. A BM25-competitive split is a sign that exact lexical anchoring remains central, not that the task is easy.

Dense Profile

Dense retrieval is the leading nDCG@10 profile for many MNanoBEIR tasks. It is most informative when the target relation depends on paraphrase, answerability, or relatedness rather than direct term overlap. This is visible in duplicate question retrieval, NQ-style passage retrieval, FEVER-style evidence selection, and some finance and scientific tasks.

Dense retrieval can still lose rare exact anchors. Entity-heavy tasks, medical terms, numeric facts, paper titles, and translated names can be underweighted if the embedding model smooths them into broader topical similarity. This makes the BM25-versus-dense comparison useful: it tells whether a task is mainly lexical, mainly semantic, or dependent on both.

Reranking Hybrid Profile

reranking_hybrid is the practical reranker-candidate view of MNanoBEIR. It often performs best when BM25 and dense retrieval find different relevant documents. ClimateFEVER, DBPedia, Touche, SCIDOCS, and several multilingual open-domain QA splits show this complementarity.

When reranking_hybrid is best by nDCG@10, the task is signaling that neither exact matching nor embedding similarity alone is sufficient. When it is not the best top-rank sorter but has stronger Recall@100, it is still useful for reranker experiments because it preserves positives that either first-stage method would otherwise drop.

Language Summary

Task Navigation

Interpretation Notes for Model Researchers

The strongest use of MNanoBEIR is controlled comparison. Because every language uses the same task grid, score differences can be read as language robustness only after accounting for the task family. A model may be excellent on translated Quora duplicates yet weak on translated finance QA or scientific related-paper retrieval. Family-level breakdowns are therefore more useful than one global average.

Pay special attention to profile changes. BM25-led rows suggest exact strings, entities, or many positives. Dense-led rows suggest paraphrase and semantic answerability. Hybrid-led rows suggest that sparse and dense retrieval recover different useful candidates. For reranker research, reranking_hybrid is often the most relevant candidate pool even when dense has the best top-rank nDCG.

Training and Leakage Notes

Useful training data should be both multilingual and task-matched: translated MS MARCO-style query-passage data for web retrieval, multilingual FEVER-style claim-evidence pairs for fact checking, duplicate-question pairs for Quora-like tasks, argument-counterargument pairs for ArguAna, and biomedical or scientific retrieval data for NFCorpus and SCIDOCS. Mixing all tasks into one generic similarity objective will blur the distinctions this group is designed to test.

Leakage risk is high because the underlying BEIR tasks are common in retrieval training mixtures. Exclude exact MNanoBEIR queries, qrels, positives, translated documents, and direct translations of evaluation text. Audit overlap with MS MARCO, NQ, FEVER, Quora, NFCorpus, SCIDOCS, SciFact, and Touche-style corpora before using them for training or synthetic-data seeding.

Source Reference Table

Metadata Summary

Task Metadata Summary