NanoMTEB-Misc / 2022_ru

Overview

2022_ru is the Russian news retrieval split of TREC 2022 NeuCLIR. Queries are Russian information needs, and documents are Russian news articles from a NeuCLIR hard-negative retrieval pool. The Nano split contains 44 queries, 8,722 documents, and 1,664 positive qrels. It is strongly multi-positive: queries have 37.82 positives on average, the median is 33, and 95.45% of queries have more than one positive. Queries average 85.57 characters, while documents average 2,448.87 characters. The task evaluates Russian topical news search over long articles and broad judged-relevant article sets.

Details

What the Original Data Measures

Overview of the TREC 2022 NeuCLIR Track defines NeuCLIR as a TREC benchmark for neural cross-language retrieval over Chinese, Persian, and Russian news. The original task used English topics to retrieve non-English news documents, with translated topic variants and pooled TREC-style relevance judgments. This Nano split uses Russian topic text against Russian documents, so it evaluates monolingual Russian retrieval under the NeuCLIR topic structure.

The task is ad hoc news search rather than question answering. A topic can have many relevant articles, and the goal is to rank useful articles about the information need early.

Observed Data Profile

The split has 44 Russian queries, 8,722 documents, and 1,664 positive judgments. Documents are long Russian news articles, often with headlines and full article bodies. Relevant pools are large: many queries have dozens of positive documents, with a maximum of 100.

Examples include Boeing 737 Max defects, NASA's first Mars helicopter flight, AI in agriculture, possible health risks from Teflon, and Russian attitudes toward Volodymyr Zelenskyy. These topics require article-level event and issue matching, not answer-span retrieval.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.3490, hit@10 of 0.8636, and recall@100 of 0.4766. Russian lexical matching is useful: named entities, product names, countries, and event phrases often overlap between topic statements and article text. However, recall is limited because long news articles and broad relevance pools create many semantically relevant candidates that do not share exact wording.

BM25 can find at least one relevant article for most queries, but it covers less than half of all positives by top 100. That makes it a useful baseline but not a complete candidate source for this multi-positive task.

Dense Evaluation Profile

Dense retrieval is strong, with nDCG@10 of 0.5807, hit@10 of 0.9318, and recall@100 of 0.6785. It substantially improves over BM25 by capturing event and topic similarity beyond exact word overlap. Dense retrieval is especially important when articles describe the same information need with different phrasing or emphasize different aspects of a news event.

The dense profile has the best recall@100 among the three candidate sources, which makes it valuable for reranking and coverage-oriented evaluation. It also indicates that Russian topical semantic matching is a central signal in this split.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile has the best nDCG@10, reaching 0.6011, with hit@10 of 0.9318 and recall@100 of 0.6130. It has lower recall than dense retrieval but better early ranking. Candidate lists contain 100 to 101 rows, with one safeguard-positive row.

This is a hybrid-favorable top-ranking case. Combining lexical anchors with dense semantics improves early precision for Russian news search, even though dense retrieval covers more total positives by rank 100.

Metric Interpretation for Model Researchers

2022_ru is hybrid-favorable for top-10 ranking and dense-favorable for top-100 coverage. BM25 is useful but loses many positives in broad news topics. Because the task is highly multi-positive, hit@10 is a coarse signal: it says whether any relevant article appears early. nDCG@10 and recall@100 are more informative for comparing ranking quality and relevant-cluster coverage.

Researchers should interpret improvements as gains in topical news retrieval, not as single-answer QA performance. A strong model should retrieve multiple articles covering the same information need.

Query and Relevance Type Tendencies

Queries are Russian topic statements, usually asking for articles about an event, issue, technology, health risk, or political attitude. Positive documents are Russian news articles relevant to that topic. Many positives may cover different facets of the same issue.

Relevance is event or issue satisfaction. Articles that mention the same named entity can still be non-relevant if they cover a different event angle.

Representative Failure Modes

BM25 can over-rank articles with shared named entities but wrong topic focus. Dense retrieval can retrieve semantically related articles that are not in the judged relevant set. Hybrid retrieval can improve early precision while missing some relevant articles that dense would include in the top 100.

Long article bodies can introduce off-topic terms, and news clusters can contain many near-duplicates or adjacent stories that make relevance judgments fine-grained.

Training Data That May Help

Useful training data includes Russian news search pairs, TREC-style Russian topic retrieval, cross-lingual English-Russian retrieval data, and hard negatives from same-topic news clusters. Training should exclude NeuCLIR evaluation topics, qrels, and article pools that overlap with the Nano split.

Synthetic data should create Russian topical search requests from news clusters and pair each request with several relevant articles. Hard negatives should share entities, countries, or broad news categories but cover a different information need.

Model Improvement Notes

Models should preserve Russian named entities and event terminology while capturing broader article semantics. Dense encoders should improve topic coverage, and hybrid rerankers should learn when lexical overlap identifies the best early article versus only a related one.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets