NanoVNMTEB / nano_nq

Overview

nano_nq is the Vietnamese NanoVNMTEB version of the Nano Natural Questions retrieval task. Natural Questions was built from real Google search questions and Wikipedia evidence, with answer annotations over long and short answer spans. In this retrieval setting, short translated questions retrieve Vietnamese-translated Wikipedia-style passages that contain the answer evidence.

The Nano split contains 200 queries, 10,000 candidate documents, and 234 positive qrels. Queries average 39.4 characters, and documents average 565.2831 characters. Most queries have one positive, though 32 queries have multiple positives. Dense retrieval is strongest on nDCG@10 and hit@10, while reranking_hybrid has the best recall@100. The task is a concise open-domain QA retrieval benchmark where a model must match the asked attribute, not merely the named entity.

Details

What the Original Data Measures

Natural Questions uses naturally occurring Google search queries and Wikipedia pages from search results. Annotators mark answer-containing spans, which makes the task closer to real open-domain QA than synthetic question writing. In retrieval form, the system must retrieve the passage that contains the answer evidence.

The Vietnamese NanoNQ variant translates the queries and evidence passages. Questions often ask for people, locations, dates, works, definitions, meanings, or relationships. Relevant passages are Wikipedia-style documents headed by an entity or topic title. The retrieval challenge is to rank the passage that answers the specific question.

Observed Data Profile

The task has 234 positives across 200 queries. The average positive count is 1.17, the median is 1, and the maximum is 3. The multi-positive rate is 16.0%. This means the benchmark is mostly single-answer evidence retrieval, with a small number of alternate evidence passages.

Queries are short and search-like. Documents are moderately long Wikipedia-style passages, often containing more context than the answer itself. A query such as a TV-character relation, a song phrase origin, or a geographic fact may share entity words with multiple passages, but only one passage contains the needed attribute.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.6095441594, hit@10 of 0.7650, and recall@100 of 0.8717948718 with a top-500 candidate set. Entity names and direct keywords help lexical retrieval, and many questions include recognizable titles or places.

The gap between BM25 and dense retrieval is large. BM25 often retrieves entity-related passages but struggles to identify the answer-bearing passage when wording changes or when the query asks for an attribute rather than an entity page. Same-entity different-attribute negatives are a major challenge.

Dense Evaluation Profile

Dense retrieval with harrier-oss-270m reaches nDCG@10 of 0.8495036532, hit@10 of 0.9200, and recall@100 of 0.9572649573. It is clearly strongest on top-rank quality. This suggests that embeddings capture the semantic relation between a short question and the answer-bearing passage more effectively than lexical overlap.

Dense retrieval is especially valuable for queries where the target attribute is implicit: who a character's father is, when a season was released, where a phrase came from, what books someone wrote, or which river is associated with a city. The model must map the question form to evidence content, not simply match terms.

Reranking Hybrid Evaluation Profile

reranking_hybrid reaches nDCG@10 of 0.7234286722, hit@10 of 0.8650, and recall@100 of 0.9914529915. The top-100 candidate pool has mean candidate count 100.005, with one safeguard-positive row and one row containing 101 candidates. Hybrid retrieval gives the best recall@100 but is below dense retrieval on top-rank metrics.

This pattern is typical for open-domain QA retrieval. Sparse signals help ensure that the positive passage is in the candidate pool, especially when entity names are exact. However, hybrid ranking can also promote same-entity distractors. Dense retrieval better orders the passage that answers the asked attribute.

Metric Interpretation for Model Researchers

Because most queries have one positive, nDCG@10 and hit@10 are central. Recall@100 is useful for candidate generation, but a downstream QA model benefits most when the answer-bearing passage is near the top. Dense retrieval is the best top-rank signal in this split.

The hybrid recall result is still important: if a reranker can recover dense-like ordering over the high-recall candidate pool, it should improve the full pipeline. Evaluation should distinguish candidate coverage from final answer-passage ranking.

Query and Relevance Type Tendencies

Queries ask about entertainment, geography, language usage, TV characters, books, rivers, release dates, and named entities. They are often short and ungrammatical in search-query style. Relevant documents are passages containing the specific answer evidence.

Relevance is attribute-specific. A passage about Bates Motel is not enough unless it identifies Dylan's father. A passage about Rome is not enough unless it contains the river relation. This makes the task stricter than broad entity retrieval.

Representative Failure Modes

BM25 can retrieve the wrong page for a shared entity or title. Dense retrieval can fail when the query is too short or when entities are ambiguous after translation. Hybrid retrieval can maximize recall while ranking a lexical distractor above the actual answer passage.

Another failure mode is answer-slot mismatch. The model may retrieve a passage about a work's cast when the query asks for release date, or a geography page when the query asks for a related river. Hard negatives should target these same-entity different-attribute cases.

Training Data That May Help

Useful training data includes official Natural Questions training examples with overlap removed, Vietnamese factoid QA over Wikipedia, non-overlapping Wikipedia question-passage retrieval pairs, and translated NQ data. Training should preserve the distinction between passage retrieval and answer extraction.

Synthetic data can generate short Vietnamese search-style questions from Wikipedia passages. It should include hard negatives that mention the same entity but answer a different attribute, because that is the main distinction retrieval must learn.

Model Improvement Notes

The main improvement direction is attribute-aware dense retrieval with high-recall hybrid candidates. Sparse retrieval should preserve entity names and titles. Dense retrieval and reranking should focus on whether the passage answers the question's requested slot.

Error analysis should classify misses by entity ambiguity, attribute mismatch, translation issue, and long-passage distraction. Top-rank precision matters more than broad topical recall for this split.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: translated VN-MTEB NanoNQ test split from GreenNode/nano-nq-vn
• Train/eval overlap audit: not_audited
• Leakage note: Exclude translated NanoNQ-VN test queries, qrels, documents, and positive passages used by this Nano split.
• Multi-positive training: multi_positive_objective
• Useful training data: official Natural Questions training examples with overlap removed, Vietnamese factoid QA over Wikipedia, non-overlapping Wikipedia question-passage retrieval pairs, translated NQ data with overlap removed