NanoMIRACL / te

Overview

NanoMIRACL / te is the Telugu split of the MIRACL-style multilingual monolingual retrieval benchmark. Telugu queries retrieve Telugu Wikipedia passages, not translated evidence. The Nano split has 200 queries, 10,000 documents, and 211 positive qrel rows. Unlike many other MIRACL Nano splits, it is almost single-positive, and many questions are entity- and census-oriented. Current diagnostics show dense retrieval as the strongest top-rank profile by a large margin, reranking_hybrid as the strongest recall profile, and BM25 as a weaker lexical baseline for this highly repetitive passage setting.

Details

What the Original Data Measures

MIRACL was introduced as a multilingual ad hoc retrieval benchmark over Wikipedia passages. Its design is monolingual: Telugu queries retrieve Telugu passages from Telugu Wikipedia. The benchmark emphasizes native-language questions, passage-level evidence, and human relevance judgments.

Telugu is one of the MIRACL languages connected to the TyDi/Mr. TyDi lineage. The MIRACL framing adds passage-level relevance judgments over a segmented Wikipedia corpus. For this split, the relevant item is a Telugu passage that contains answer evidence, not a translated English passage or a short answer.

Observed Data Profile

The Nano split contains 200 queries, 10,000 documents, and 211 positive qrel rows. Positives per query average 1.05, with a minimum of 1, a median of 1, and a maximum of 3. There are only 9 multi-positive queries, representing 4.5 percent of the split. Queries average 38.41 characters, while documents average 409.03 characters.

The examples are strongly entity- and attribute-oriented. Many ask about 2011 census values, village area, male population, number of houses, pin codes, birthplaces, founders, scientific names, religious sites, institutions, and definitions. Telugu village pages often repeat similar census-style prose, so the task requires exact entity disambiguation as well as attribute matching.

BM25 Evaluation Profile

The dataset-provided BM25 candidate subset contains 500 candidates per query and achieves nDCG@10 = 0.5292, hit@10 = 0.6400, and recall@100 = 0.8768. BM25 is useful when the exact village name, entity name, or attribute phrase appears in both query and passage. It can exploit repeated Telugu terms for area, population, pin code, and scientific name.

The sparse profile is weak because many candidate passages are near-duplicates in structure. Village pages can share the same census phrases, distances, population formulas, caste-count language, and pin-code patterns. BM25 may rank a passage with the right attribute words above the exact village or entity page.

Dense Evaluation Profile

The dense harrier_oss_v1_270m candidate subset contains 500 candidates per query and achieves nDCG@10 = 0.8720, hit@10 = 0.9150, and recall@100 = 0.9194. Dense retrieval is the strongest observed profile by nDCG@10 and hit@10. It greatly improves top-rank ordering by connecting the query's entity and attribute to the answer-bearing passage.

This is a clear dense-retrieval advantage. The model must distinguish whether a question asks for area, population, male count, number of houses, pin code, founder, birthplace, or scientific name, even when many passages share the same template. Dense retrieval ranks the exact evidence much better than BM25.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset contains mostly 100 candidates per query, with three queries using a rank-101 safeguard row. It achieves nDCG@10 = 0.6953, hit@10 = 0.8650, and recall@100 = 0.9810. Hybrid retrieval is below dense retrieval in top-rank quality, but it has the best positive coverage.

This means hybrid search is useful primarily as a candidate generator for this Telugu split. BM25 contributes exact entity and attribute phrases, while dense retrieval contributes semantic and structural matching. The hybrid candidate pool retains more positives for reranking, but dense retrieval alone ranks the top evidence more accurately.

Metric Interpretation for Model Researchers

This task is close to single-positive: only 4.5 percent of queries have more than one positive passage. Hit@10 measures whether the labeled passage appears near the top. nDCG@10 is especially sensitive to rank position because most queries have one relevant target. recall@100 measures whether the target survives for reranking.

The Telugu pattern is diagnostic. Dense retrieval is the best top-rank model, BM25 struggles with repeated templates, and reranking_hybrid is the best coverage source. Researchers should evaluate exact entity disambiguation and attribute binding, not only general semantic similarity.

Query and Relevance Type Tendencies

Queries frequently ask about village statistics and local attributes: area, population, male count, number of houses, pin code, census values, and location. Other questions cover scientific names, religious or institutional facts, authorship, birthplaces, founders, and historical definitions.

Relevant documents are Telugu Wikipedia passages with title context and answer-bearing prose. The task rewards exact Telugu-script entity preservation, template-aware passage selection, and relation matching between the query attribute and the passage field.

Representative Failure Modes

BM25 can retrieve the wrong village page because many passages share the same గ్రామ విస్తీర్ణం, census, population, or pin-code wording. A pumpkin scientific-name query can retrieve other plant pages with శాస్త్రీయ నామం before the pumpkin passage. Pin-code questions can retrieve nearby or similarly formatted village pages. Organization or authorship questions can retrieve related cultural or institutional pages that share terms but lack the answer.

Dense retrieval can still miss an exact entity when similar village templates are very close semantically. Hybrid retrieval improves recall but still requires reranking to select the precise village or attribute-bearing passage.

Training Data That May Help

Useful training data includes non-overlapping MIRACL Telugu training data, Telugu Wikipedia question-to-passage retrieval pairs, Telugu open-domain QA evidence retrieval datasets, and synthetic village-statistic retrieval pairs from non-evaluation Telugu pages. Hard negatives should include near-duplicate village pages that differ only by name, area, population, pin code, or census line.

Synthetic data can help when it creates Telugu Wikipedia-style passages with titles, aliases, census prose, areas, population counts, pin codes, founders, birthplaces, scientific names, and institutional descriptions. Generated questions should include 2011 census, village area, male count, number of houses, pin code, who, where, and scientific-name forms with exact entity names. Comparable evaluation should exclude upstream development/test data or other MIRACL-derived examples likely to overlap with this Nano split.

Model Improvement Notes

Dense retrievers should preserve their strong top-rank advantage while improving coverage toward the hybrid profile. Sparse systems need better Telugu tokenization and entity weighting for repeated census templates. Rerankers should explicitly bind village name and requested attribute, rather than rewarding shared boilerplate.

For hybrid systems, NanoMIRACL / te supports reranking_hybrid as a high- recall candidate stage, followed by a reranker specialized for exact entity and field matching. Dense retrieval sets a strong top-rank target for this split.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: unknown
• Train/eval overlap audit: not_audited
• Leakage note: prefer excluding upstream development/test data or other MIRACL-derived data likely to overlap with the NanoMIRACL evaluation questions and passages
• Multi-positive training: single_positive_question_document_focus
• Useful training data: non-overlapping MIRACL Telugu train split data, Telugu Wikipedia question-to-passage retrieval pairs, Telugu open-domain QA evidence retrieval datasets, synthetic village-statistic retrieval pairs from non-evaluation Telugu pages