NanoMLDR / en

Overview

NanoMLDR / en is the English split of NanoMLDR, a multilingual long-document retrieval benchmark derived from MLDR. English paragraph-grounded questions retrieve full English articles, and the answer-bearing paragraph may be a small part of an extremely long document. The Nano split has 200 queries, 10,000 documents, and 200 positive qrel rows, with exactly one positive document per query. Current diagnostics show BM25 as the strongest top-rank profile, reranking_hybrid as tied with BM25 on recall@100, and dense retrieval as much weaker for full-document matching.

Details

What the Original Data Measures

MLDR was introduced with the M3-Embedding work as a multilingual long-document retrieval benchmark. The dataset card describes a construction process in which long documents are sampled, a paragraph is selected, and a specific question is generated from that paragraph. The positive is the full article containing the answer-bearing paragraph.

For English, this creates a long-document retrieval task rather than a passage retrieval task. A model must retrieve the entire article from a question that may only correspond to a small local section. The benchmark therefore stresses long-document indexing, paragraph-to-document matching, and robustness to documents whose main topic is broader than the query.

Observed Data Profile

The Nano split contains 200 queries, 10,000 documents, and 200 positive qrel rows. Every query has exactly one positive document. Queries average 64.06 characters, while documents average 27,991.90 characters. This is one of the longest NanoMLDR splits and makes full-document representation difficult.

Observed questions cover rare-earth elements, jazz reissue labels, PCR limitations, UN Security Council Resolution 242, composite bows, protein engineering, aviation events, aquarium conservation, taxonomy, constitutional law, and other article-level topics. The question is specific, but the positive document can contain tens of thousands of characters of surrounding context.

BM25 Evaluation Profile

The dataset-provided BM25 candidate subset contains 500 candidates per query and achieves nDCG@10 = 0.7254, hit@10 = 0.8300, and recall@100 = 0.9300. BM25 is the strongest observed top-rank profile. Long English articles preserve many rare terms, names, dates, technical phrases, and paragraph-specific expressions from the generated question.

This is a setting where lexical matching is highly valuable. If the question is grounded in a paragraph about PCR, a UN resolution, a bow design, or a protein engineering method, the full article often contains exact anchor terms that BM25 can use even when the answer paragraph is buried deep in the text.

Dense Evaluation Profile

The dense harrier_oss_v1_270m candidate subset contains 500 candidates per query and achieves nDCG@10 = 0.4611, hit@10 = 0.5400, and recall@100 = 0.7200. Dense retrieval is much weaker than BM25. A single dense representation of a very long article can dilute the paragraph-level evidence needed for retrieval.

Dense similarity may capture broad article themes but miss the precise local fact that generated the question. This is especially problematic when several long articles share the same high-level domain, such as biology, law, military history, or music history, but only one contains the answer-bearing paragraph.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset contains mostly 100 candidates per query, with 14 queries using a rank-101 safeguard row. It achieves nDCG@10 = 0.5916, hit@10 = 0.7250, and recall@100 = 0.9300. Hybrid retrieval matches BM25 on recall@100 but remains below BM25 on top-rank metrics.

This profile suggests that dense candidates add some useful coverage but do not improve the observed top ranking over BM25. For English MLDR, hybrid search is most useful as a candidate pool for reranking, while BM25 remains the strongest base ranker in the current diagnostics.

Metric Interpretation for Model Researchers

This task is single-positive: each query has exactly one relevant long document. Hit@10 measures whether that document appears near the top. nDCG@10 is sensitive to the exact rank of the positive document, and recall@100 measures whether it survives for reranking.

The English MLDR pattern is a direct test of long-document retrieval. Strong performance requires preserving paragraph-specific evidence inside long documents. Single-vector dense retrieval is not enough here; systems should consider chunked indexing, late interaction, multi-vector representations, or paragraph-aware document aggregation.

Query and Relevance Type Tendencies

Queries are English paragraph-grounded fact questions about people, methods, limitations, legal or diplomatic statements, technical steps, historical records, and scientific properties. They are often specific but not necessarily aligned with the article title.

Relevant documents are very long English articles containing the answer-bearing paragraph. The task rewards exact rare-term matching, entity preservation, paragraph-to-document linking, and robust ranking among topically adjacent long articles.

Representative Failure Modes

Dense retrieval can return a thematically related article while missing the one paragraph that actually contains the answer. Long documents about similar scientific, legal, historical, or cultural subjects can collapse into nearby embedding space. BM25 can fail when the question paraphrases the paragraph or when many articles share the same rare entity or technical term.

Hybrid retrieval can include the positive but still rank another broad article higher. Rerankers should inspect chunks or paragraphs rather than scoring only the full article text as one unit.

Training Data That May Help

Useful training data includes English long-document QA retrieval pairs, English Wikipedia article retrieval, NarrativeQA-style long-document evidence retrieval, and entity-sharing article hard negatives. Training should include cases where a small paragraph determines the relevance of a long article.

Synthetic data can help when it samples paragraphs from long English encyclopedic articles, generates grounded fact questions, and uses the full article as the positive. Hard negatives should be topically adjacent full articles that share names, dates, methods, or scientific terms but do not contain the answer-bearing paragraph.

Model Improvement Notes

Dense retrievers should move beyond single-vector full-article encoding. Chunked retrieval, paragraph-aware pooling, late interaction, or multi-vector document representations are better matched to English MLDR. Sparse systems should preserve rare lexical anchors while reducing over-ranking from repeated boilerplate or broad topic terms.

For hybrid systems, NanoMLDR / en suggests starting from BM25 as a strong candidate generator and adding dense signals carefully. The current hybrid profile preserves recall but does not beat BM25 top-rank quality, so reranking must be evaluated against the sparse baseline.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: MLDR English split
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoMLDR en queries, qrels, and positive documents
• Multi-positive training: single_positive
• Useful training data: English long-document QA retrieval pairs, English Wikipedia article retrieval, NarrativeQA-style long-document evidence retrieval, entity-sharing article hard negatives