NanoCMTEB / mmarco

Overview

NanoCMTEB mmarco is a Chinese passage retrieval task based on mMARCO, the multilingual version of MS MARCO passage ranking. Queries are short Chinese fact-seeking information needs, and documents are translated web passages. The task measures whether retrieval systems can rank answer-bearing Chinese passages while handling translation artifacts, named entities, and short-query ambiguity.

Details

What the Original Data Measures

mMARCO translates MS MARCO passage-ranking data into multiple languages, including Chinese. The relevance structure comes from MS MARCO-style web information needs, while the surface text is machine translated. C-MTEB includes MMarcoRetrieval as part of its Chinese retrieval evaluation.

This task is therefore not identical to native Chinese web search. It tests multilingual and translated-passage retrieval, where English-origin entities, brands, names, and phrasing may remain visible in Chinese text.

Observed Data Profile

The task contains 200 queries, 10,000 documents, and 212 relevance judgments. It is mostly single-positive: there are 1.06 positives per query on average, a minimum of 1, a median of 1.0, a maximum of 2, and 12 multi-positive queries, or 6.00% of the set.

Queries average 10.44 Chinese characters, and documents average 113.91 characters. Examples include historical effects, biology and kidney function, actors and film roles, consumer electronics, word meanings, animal facts, and celebrity information.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.6795, hit@10 of 0.8050, and recall@100 of 0.9104 using the top-500 BM25 candidate subset. This is a strong lexical baseline because many fact-seeking queries contain named entities, technical terms, or translated answer terms that appear in the positive passage.

BM25 can struggle when translation choices differ, when the query is paraphrased, or when entity-sharing negatives answer a different fact. It is strong but not the best ranking profile.

Dense Evaluation Profile

The dense harrier-oss-270m run reaches nDCG@10 of 0.8859, hit@10 of 0.9350, and recall@100 of 0.9717. Dense retrieval is the strongest top-ranking profile by a wide margin. It improves substantially over BM25 in nDCG@10 and hit@10.

This indicates that embedding similarity is effective for translated MS MARCO-style retrieval. Dense retrieval can connect short Chinese queries to answer-bearing passages even when wording differs or translation artifacts appear.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate set reaches nDCG@10 of 0.7984, hit@10 of 0.9000, and recall@100 of 0.9858. It uses a top-100 candidate range with an optional rank-101 safeguard; this task has 3 safeguard rows, candidate counts from 100 to 101, and a mean of 100.02 candidates.

Hybrid retrieval has the best recall@100, but dense retrieval remains best for top-10 ranking. The hybrid pool is valuable for downstream reranking, while dense is the strongest observed first-stage ranker.

Metric Interpretation for Model Researchers

This task is dense-favorable with a hybrid coverage advantage. BM25 is already strong because many translated facts preserve lexical anchors, but dense retrieval gives much better first-page ordering. Reranking_hybrid exposes slightly more positives in the top 100.

Researchers should account for translation artifacts. A model that performs well may be handling both Chinese semantics and English-origin entity traces. Evaluation should focus on answer-bearing relevance, not only entity overlap.

Query and Relevance Type Tendencies

Queries ask about historical effects, biological mechanisms, celebrity roles, device setup, word meanings, consumer facts, travel advice, and simple factual knowledge. Positive documents are short translated passages that contain the answer.

The relevance relation is answer containment. A passage is positive if it satisfies the information need, often with a concise explanation or factual statement.

Representative Failure Modes

Likely failures include retrieving a passage with the same entity but the wrong fact, missing a passage due to translation variation, over-ranking literal term matches that do not answer the query, and confusing ambiguous short queries.

BM25 is vulnerable to translated lexical mismatch and entity-sharing distractors. Dense retrieval can over-generalize topic similarity. Hybrid retrieval improves coverage but can lose some dense top-rank precision.

Training Data That May Help

Useful training data includes non-overlapping mMARCO Chinese query-passage pairs, multilingual MS MARCO passage ranking data, Chinese fact-seeking QA retrieval pairs, and translated entity-sharing hard negatives.

Synthetic data can translate or generate fact-seeking web passages and Chinese queries over them. Hard negatives should share entity names, units, or question types while answering a different fact.

Model Improvement Notes

Strong systems should handle Chinese short queries, translated passage phrasing, and named-entity preservation. Dense retrieval is the strongest observed first-stage method, while hybrid retrieval is useful when recall@100 matters.

The task is useful for evaluating multilingual retrieval models on Chinese MS MARCO-style passage ranking, especially when translation quality and entity handling influence retrieval effectiveness.

Example Data

Source Reference Table

Representative query and positive source snippets:

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: MMarcoRetrieval dev
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoCMTEB mmarco queries, qrels, and translated passages
• Multi-positive training: preserve_small_number_of_multiple_positives
• Useful training data: non-overlapping mMARCO Chinese pairs, multilingual MS MARCO passage ranking data, Chinese fact-seeking QA retrieval pairs, translated entity-sharing hard negatives