NanoMTEB-Misc / 2022_zh

Overview

2022_zh is the Chinese news retrieval split of TREC 2022 NeuCLIR. Queries are Chinese topic statements, and documents are Chinese news articles from a NeuCLIR hard-negative retrieval pool. The Nano split contains 47 queries, 10,000 documents, and 1,643 positive qrels. It is strongly multi-positive: queries have 34.96 positives on average, the median is 18, and 97.87% of queries have more than one positive. Queries average 24.00 characters, while documents average 1,107.60 characters. The task evaluates Chinese topical news search over broad event and issue clusters, not single-answer passage retrieval.

Details

What the Original Data Measures

Overview of the TREC 2022 NeuCLIR Track describes NeuCLIR as a neural cross-language retrieval benchmark over Chinese, Persian, and Russian Common Crawl news. The original task used English topics to retrieve target-language articles, with monolingual translated topics also available for reference. This Nano split uses Chinese queries and Chinese documents, preserving the TREC-style information needs and pooled relevance judgments.

The retrieval target is any judged relevant article for the topic. A query can have many relevant articles, often covering different facets of an event, public-health topic, technology issue, or political development.

Observed Data Profile

The split has 47 Chinese queries, 10,000 documents, and 1,643 positive judgments. Documents are Chinese news articles with headlines and body text. Queries are short compared with Persian and Russian NeuCLIR topics, which makes query-document matching more dependent on semantic topic alignment and named entity handling.

Examples include Iran's internet shutdown during protests, domestic COVID vaccine production in Iran, AI in agriculture, discrimination against people with AIDS in China, and myopia among Chinese students. These are topical search requests rather than factoid questions.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.2931, hit@10 of 0.7872, and recall@100 of 0.2958. It can often find at least one relevant article when a query contains distinctive entities or terms, but it covers a small share of the positive set by rank 100. Short Chinese queries and broad news topics limit the effectiveness of exact term matching.

The low recall@100 is the key BM25 weakness. Even with many positives per query, the sparse candidate list misses many relevant articles because they use different wording, locations, or event framing.

Dense Evaluation Profile

Dense retrieval is the strongest profile, with nDCG@10 of 0.5101, hit@10 of 0.9149, and recall@100 of 0.6245. The dense model captures Chinese news topic semantics much better than BM25, especially when relevant articles describe the same issue without repeating the exact topic wording.

For researchers, this split is a strong diagnostic of Chinese event-level semantic retrieval. A model must map compact Chinese topic statements to long news articles and recover multiple related articles in a broad cluster.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile reaches nDCG@10 of 0.4072, hit@10 of 0.8936, and recall@100 of 0.4918. It improves over BM25 but remains below dense retrieval in both early ranking and coverage. Candidate lists contain 100 to 101 entries, with two safeguard-positive rows.

This pattern indicates that lexical anchors help, but the dominant signal is semantic topic matching. Hybrid search is more robust than BM25 alone, yet the dense candidate set is the better starting point for this Chinese NeuCLIR split.

Metric Interpretation for Model Researchers

2022_zh is dense-favorable. BM25 retrieves some obvious matches, dense retrieval provides the best ranking and coverage, and reranking_hybrid lands between them. Because nearly all queries have many positives, hit@10 is less informative than nDCG@10 and recall@100: finding one article is useful, but the benchmark also rewards broader coverage of relevant news articles.

Top-10 quality reflects whether the model ranks strongly relevant articles early, while recall@100 reflects whether it can expose enough of the topic cluster for reranking or downstream analysis.

Query and Relevance Type Tendencies

Queries are Chinese topical information needs about public events, technology, health, agriculture, politics, or social issues. Positive documents are Chinese news articles relevant to the topic. Relevance is broad but judged: articles must satisfy the information need, not simply mention the same entity.

The task rewards retrieval over news clusters, including multiple articles that cover the same issue from different angles.

Representative Failure Modes

BM25 can miss relevant articles that use different wording or focus on an adjacent aspect of the topic. Dense retrieval can retrieve semantically related but unjudged or off-angle articles. Hybrid retrieval can improve lexical precision while still missing much of the positive set.

Short queries can under-specify the desired angle, and long news articles can contain many terms that distract sparse retrieval.

Training Data That May Help

Useful training data includes Chinese news retrieval pairs, Chinese TREC-style topic retrieval, multilingual CLIR data, and hard negatives from same-topic news clusters. Training should exclude NeuCLIR evaluation topics, qrels, and article pools that overlap with this Nano split.

Synthetic data should generate Chinese search topics from related news clusters and assign several relevant articles per topic. Hard negatives should share entities or event categories but differ in relation, impact, or location.

Model Improvement Notes

Models should handle Chinese segmentation, named entities, and event semantics. Dense encoders should be trained for multi-positive topical coverage. Rerankers should distinguish articles that truly answer the information need from articles that merely share high-level news vocabulary.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets