NanoVNMTEB

Overview

NanoVNMTEB is the Nano task group for VN-MTEB retrieval. It contains Vietnamese retrieval versions of widely used MTEB and BEIR-style tasks, including duplicate question retrieval, fact-checking evidence retrieval, web search, natural question answering, finance QA, argument retrieval, biomedical retrieval, and scientific-paper retrieval. The group evaluates Vietnamese retrieval quality and robustness to translated benchmark artifacts.

The group contains 4,768 queries, 247,475 task-local documents, and 24,671 positive qrel rows across 26 tasks. Most tasks are Vietnamese, while nfcorpus_vn is marked multilingual because biomedical terminology and translation artifacts cross language boundaries. The group is large enough that aggregate scores can hide very different retrieval relations.

What This Group Measures

VN-MTEB translates and filters English MTEB datasets into Vietnamese while preserving named entities, numbers, links, special characters, fluency, and meaning. NanoVNMTEB focuses on the retrieval family from that benchmark. A high score means that a model can preserve the original retrieval relation after Vietnamese translation, whether the relation is duplicate intent, evidence support, web answerability, scientific relatedness, argument stance, or biomedical relevance.

The group should not be read as one native Vietnamese corpus. Many tasks inherit their semantics from English benchmark sources. This makes it valuable for testing multilingual and translation-robust retrievers, but it also means that source-task shape matters as much as Vietnamese language quality.

Task Families

• Duplicate and paraphrase retrieval: ten CQADupStack splits plus quora_vn retrieve duplicate or equivalent questions.
• Fact-checking and evidence retrieval: fever_vn, nano_fever, climate_fever_vn, and sci_fact_vn retrieve evidence for claims.
• Open-domain and web QA retrieval: msmarco_vn, nq_vn, nano_nq, and hotpot_qa_vn retrieve answer-bearing passages.
• Argument retrieval: argu_ana_vn retrieves counterarguments and touche2020_vn retrieves argumentative passages.
• Domain retrieval: fi_qa2018_vn, nfcorpus_vn, and treccovid_vn cover finance, biomedical literature, and COVID-19 evidence.
• Entity and scholarly retrieval: dbpedia_vn retrieves entity articles, and scidocs_vn retrieves related scientific papers.

Dataset Shape

Most splits have 200 queries and 10,000 candidate documents. Smaller exceptions include sci_fact_vn, nfcorpus_vn, touche2020_vn, and treccovid_vn. Positive density varies sharply. ArguAna is single-positive. Many FEVER, NQ, and MS MARCO-style tasks are close to single-positive. DBpedia, NFCorpus, Touché, TREC-COVID, and several duplicate-question tasks are strongly multi-positive.

The group average is 5.17 positives per query, but that average is driven by large relevance sets in DBpedia, NFCorpus, Touché, and TREC-COVID. The median task behavior is closer to one or two positives per query. Query length also varies: argu_ana_vn uses long translated debate arguments, while MS MARCO, NQ, and many CQADupStack queries are short search or question strings.

Retrieval Behavior

BM25 Profile

BM25 is best for quora_vn and remains strong on entity-heavy and fact-like tasks. fever_vn, nano_fever, hotpot_qa_vn, msmarco_vn, and quora_vn all score high because translated named entities, titles, and short duplicate phrases often preserve lexical overlap. BM25 also works reasonably on many CQADupStack domains when technical terms, code fragments, product names, or StackExchange terminology survive translation.

BM25 is weak on tasks where the source relevance relation is not lexical. scidocs_vn has low nDCG@10 because related scientific papers may not share title words. climate_fever_vn, argu_ana_vn, and several CQADupStack domains also require evidence, stance, or duplicate intent beyond topical overlap. treccovid_vn has many positives, but BM25 nDCG@10 is modest because broad COVID terminology does not rank the best judged literature records early.

Dense Profile

Dense retrieval with harrier-oss-270m is the strongest group-level profile. It is best for most tasks, including argument retrieval, climate evidence, DBpedia, FEVER, MS MARCO, NQ, SciFact, and many duplicate-question domains. The large gains on msmarco_vn, nano_nq, nq_vn, and fever_vn show that Vietnamese embedding similarity helps connect translated questions and claims to answer-bearing or evidence passages.

Dense is not always best. quora_vn slightly favors hybrid, and some technical duplicate tasks favor hybrid because exact tokens and semantic intent are both important. Still, dense retrieval is the main profile for VN-MTEB-style Vietnamese translation robustness.

Reranking Hybrid Profile

The reranking hybrid profile is best for several tasks where exact tokens and semantic relatedness both matter: cqadupstack_mathematica_vn, cqadupstack_stats_vn, cqadupstack_tex_vn, cqadupstack_wordpress_vn, fi_qa2018_vn, nfcorpus_vn, quora_vn, scidocs_vn, and touche2020_vn. These tasks often contain technical terms, formulas, biomedical terminology, argument terms, or duplicate clusters where sparse and dense signals recover complementary candidates.

Hybrid has the best group-level recall@100, which is important for multi-positive tasks. It is not the best nDCG@10 profile overall because dense ranking is stronger on many translated QA and evidence tasks. For Vietnamese retrieval systems, this suggests a practical split: dense is a strong default, while hybrid is valuable for technical duplicates, biomedical retrieval, argument retrieval, and candidate generation.

Task Summary

Interpretation Notes for Model Researchers

NanoVNMTEB is a translation-robustness and task-shape benchmark. Dense retrieval dominates many translated QA and evidence tasks, suggesting that Vietnamese semantic matching is critical. Hybrid retrieval is valuable for technical duplicate clusters, biomedical and scholarly retrieval, finance QA, and argument retrieval, where exact terms and semantic relatedness both matter.

The group should be analyzed by source family. Improvements on FEVER, MS MARCO, or NQ do not necessarily imply improvements on CQADupStack, SciDocs, NFCorpus, or Touché. Multi-positive tasks such as DBpedia, NFCorpus, Touché, and TREC-COVID should be inspected with recall and ranking behavior, not only one top-hit metric.

Training and Leakage Notes

Useful training data includes Vietnamese duplicate-question pairs, Vietnamese Wikipedia QA, translated or native claim-evidence pairs, finance QA, argument retrieval, biomedical retrieval, scientific related-paper retrieval, and Vietnamese web passage retrieval. Technical tasks should preserve code snippets, math or TeX fragments, URLs, product names, and domain-specific terminology.

Leakage control should exclude NanoVNMTEB evaluation queries, qrels, positive documents, duplicate clusters, and translated variants of common benchmark test examples. Overlap audits are especially important for MS MARCO, FEVER, NQ, Quora, CQADupStack, TREC-COVID, and SciDocs because these sources often appear in multilingual or synthetic training mixtures.

Source Reference Table

Metadata Summary

Task Metadata Summary