NanoFaMTEB-v2 / wikipedia_multilingual_fa

Overview

wikipedia_multilingual_fa is a Persian Wikipedia retrieval task in NanoFaMTEB-v2. The queries are synthetic Persian information-seeking questions, and the documents are Persian Wikipedia-derived passages.

This task evaluates clean encyclopedia passage retrieval. The query and target passage usually have strong topical alignment, but the model still has to retrieve the single answer-bearing passage from a 10,000-document pool.

Details

What the Original Data Measures

FaMTEB includes Wikipedia retrieval among the Persian retrieval datasets. The source metadata says this task derives from Cohere's wikipedia-2023-11 data and contains synthetically generated queries. The public task is exposed through mteb/WikipediaRetrievalMultilingual, and MTEB provides the retrieval evaluation framework.

The benchmark measures whether a retriever can match a generated natural-language query to the source Wikipedia passage. This is useful for evaluating encyclopedia RAG retrieval in Persian and multilingual settings.

Observed Data Profile

This Nano split contains 200 queries, 10,000 documents, and 200 positive qrels. Each query has exactly one positive. Queries average 49.17 characters, and documents average 352.93 characters.

Observed examples ask about affective reactions before cognitive processing, architectural features of Taqavi School in Gorgan, low-tillage agriculture in the Midwestern United States, a spelling change from "Rudmehjan" to "Rudmajan", and the first chapters of Genesis. Positive documents are direct encyclopedia passages.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.8934, hit@10 of 0.9600, and recall@100 of 0.9800 with a top-500 candidate pool. This is a very strong lexical profile.

Synthetic queries often reuse entity names, technical terms, or distinctive phrases from the source passage. BM25 therefore finds most positives. Its errors are likely cases where a generated query paraphrases the passage or where several documents share the same entity or topic.

Dense Evaluation Profile

The dense harrier-oss-270m profile reaches nDCG@10 of 0.9007, hit@10 of 0.9550, and recall@100 of 0.9750. Dense retrieval is slightly strongest on nDCG@10, while BM25 is slightly stronger on hit and recall.

This pattern suggests that semantic matching helps order the best answer passage near the top, but exact lexical matching remains highly reliable. Dense retrieval is useful when the synthetic question is phrased differently from the passage, especially for explanatory or definitional content.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset reaches nDCG@10 of 0.8958, hit@10 of 0.9750, and recall@100 of 0.9900. It uses 100 candidates per query, with two rank-101 safeguard positives.

Hybrid retrieval gives the best hit@10 and recall@100, while dense retrieval slightly leads nDCG@10. For reranking, the hybrid pool is the most reliable because it combines lexical entity coverage with semantic passage matching and nearly always includes the labeled positive.

Metric Interpretation for Model Researchers

wikipedia_multilingual_fa is a high-baseline encyclopedia retrieval task. Because each query has exactly one positive, nDCG@10 and hit@10 directly indicate whether the answer passage is found near the top.

The three profiles are close, so small differences should be interpreted carefully. Dense retrieval is best for direct ranking quality, BM25 is highly competitive, and hybrid retrieval is best for candidate coverage. This makes the task useful as a sanity check for Persian Wikipedia retrieval rather than as the hardest benchmark in the set.

Query and Relevance Type Tendencies

Queries are Persian synthetic questions generated from Wikipedia passages. They often ask about definitions, reasons, historical facts, geographic or architectural details, and procedural descriptions. Documents are concise encyclopedia passages.

The relevance relation is direct source-passage matching. A positive document contains the fact or explanation that motivated the generated query.

Representative Failure Modes

BM25 may miss paraphrased generated questions that do not repeat the passage's distinctive words. Dense retrieval may retrieve a semantically related passage about the same entity but not the exact source passage. Hybrid retrieval reduces these failures, but single-positive labels make near-duplicate encyclopedia passages difficult.

Another risk is overestimating generalization from synthetic query style. A model may perform well on generated Wikipedia questions while being less robust on real user queries with misspellings or incomplete context.

Training Data That May Help

Useful training data includes Persian Wikipedia query-passage retrieval, synthetic query generation over encyclopedia passages, multilingual Wikipedia retrieval, and hard negatives sharing the same entity or topic but omitting the requested fact.

Training should exclude this split's generated queries and positive passages.

Model Improvement Notes

Improving this task requires reliable Persian encyclopedia retrieval. Models should preserve entity names and technical terms while supporting paraphrased questions and explanatory relations.

For reranking, the main goal is selecting the exact answer-bearing passage among related Wikipedia passages. Because candidate coverage is already high, gains mostly come from fine-grained top-rank ordering.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets