NanoMTEB-Misc / 2022_fa

Overview

2022_fa is the Persian news retrieval split of the TREC 2022 NeuCLIR task. Queries are Persian information needs, and documents are Persian news articles drawn from a NeuCLIR hard-negative retrieval pool. The Nano split contains 45 queries, 8,882 documents, and 1,131 positive qrels. It is highly multi-positive: queries have 25.13 positives on average, the median is 12, and 93.33% of queries have more than one positive. Queries average 83.13 characters, while documents average 2,818.76 characters. This task is useful for evaluating Persian topical news retrieval, broad event coverage, and ranking over many relevant long-form articles.

Details

What the Original Data Measures

Overview of the TREC 2022 NeuCLIR Track describes NeuCLIR as a neural cross-language information retrieval benchmark over Chinese, Persian, and Russian news. The original TREC setting used English topics to retrieve documents in the target language, with translated topic variants and pooled relevance judgments. This Nano split uses Persian topic text against Persian documents, so it measures monolingual Persian news retrieval within the NeuCLIR topic and judgment structure.

NeuCLIR topics are not factoid questions. They describe information needs such as finding articles about domestic vaccine production, AI in agriculture, or health risks. A relevant set can include many articles covering the same event or issue from different angles.

Observed Data Profile

The split has 45 Persian queries, 8,882 documents, and 1,131 positive judgments. Documents are long Persian news articles with headlines and article bodies. The large positive pools make this a many-relevant-documents ranking task rather than a single-answer retrieval task.

Examples include articles about calcium-rich vegetables, Iran's COVIran Barekat vaccine, AI in agriculture, China's 5G role in Russia, and myopia among Chinese students. Queries are user information needs, often phrased as "I am looking for articles..." rather than short keyword queries.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.2600, hit@10 of 0.5778, and recall@100 of 0.5208. Lexical matching helps when topic terms, named entities, or domain phrases are repeated in the article text. However, BM25 is weak at top-10 ranking because the relevant set is broad and many long news articles share event vocabulary without necessarily being judged relevant.

The recall@100 value also shows that BM25 misses many positives even though there are many relevant documents per query. Sparse matching is not enough for semantic event coverage across long Persian news articles.

Dense Evaluation Profile

Dense retrieval is the strongest profile, with nDCG@10 of 0.4915, hit@10 of 0.9556, and recall@100 of 0.6897. The dense model is much better at connecting Persian information needs to relevant article semantics, especially when the article wording differs from the topic statement or covers the same issue from a different angle.

This task is therefore a useful Persian dense-retrieval diagnostic. Improvements should reflect better topical and event-level semantic matching, not only keyword overlap. The long document length also rewards models that can represent article-level meaning beyond the headline.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile reaches nDCG@10 of 0.4138, hit@10 of 0.8444, and recall@100 of 0.6490. It improves substantially over BM25 and approaches dense recall, but dense retrieval remains stronger at top-10 and top-100. The hybrid lists contain 100 to 101 candidates, with one safeguard-positive row.

This pattern suggests that the lexical component adds useful anchors, but the dominant signal is semantic news-topic similarity. Hybrid search is a reasonable reranking pool, but dense-only candidates are cleaner for this split.

Metric Interpretation for Model Researchers

2022_fa is dense-favorable. BM25 is limited by long documents and broad topic-event relevance, dense retrieval is strongest, and reranking_hybrid sits between them. Because the task is highly multi-positive, hit@10 only tells whether any relevant article appears early; recall@100 is important for how much of the relevant event cluster is exposed.

nDCG@10 measures whether highly useful relevant articles are ranked early. Models should not be judged as single-answer retrievers here: many documents may be relevant to the same information need.

Query and Relevance Type Tendencies

Queries are Persian topical news information needs. Positive documents are Persian news articles that satisfy the topic, often with multiple relevant articles per query. Relevance can depend on event angle, entity, country, policy issue, or health/science topic.

The task rewards broad topical recall and strong early ranking across news clusters. It is closer to ad hoc news search than QA retrieval.

Representative Failure Modes

BM25 can over-rank articles with shared named entities or repeated topic words but the wrong event angle. Dense retrieval can retrieve semantically related articles that are not judged relevant or miss narrow named-entity constraints. Hybrid retrieval can improve coverage while still mixing near-topic articles with true positives.

Long article bodies can dilute both sparse and dense signals, especially when the relevant information is concentrated in a headline or a few paragraphs.

Training Data That May Help

Useful training data includes Persian news retrieval pairs, NeuCLIR-style topic-document pairs, multilingual CLIR corpora, and hard negatives from the same event, entity, or news category. Training should avoid NeuCLIR evaluation topics, qrels, and article pools that overlap with the Nano split.

Synthetic data should generate Persian information needs from non-evaluation news clusters and pair each topic with several relevant articles. Hard negatives should share key entities but cover a different angle or event.

Model Improvement Notes

Models should represent Persian topical intent, named entities, and event semantics across long news articles. Dense encoders should be trained on multi-positive news clusters. Rerankers should compare whether an article actually satisfies the topic, not only whether it shares headline terms.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets