MNanoBEIR / NanoBEIR-ko / NanoNQ

Overview

NanoBEIR-ko__NanoNQ is the Korean NanoBEIR version of Natural Questions, an open-domain question answering retrieval benchmark based on real Google search questions and Wikipedia evidence. The task uses Korean translated questions as queries and asks a retriever to rank Korean translated Wikipedia passages that contain answer evidence. The Nano split contains 50 queries, 5,035 documents, and 57 positive qrels. Most queries have one positive passage, while 7 queries have two. Dense retrieval is strongest for the top ranks, while hybrid retrieval provides the best candidate coverage.

Details

What the Original Data Measures

Natural Questions introduced real information-seeking questions paired with Wikipedia answers and annotations. BEIR uses NQ as open-domain QA retrieval: the system must retrieve passages that contain answer evidence before a downstream reader can answer the question. In this Korean NanoBEIR version, translated questions are matched against translated Wikipedia passages, testing short question semantics, entity matching, and answer-context retrieval.

Observed Data Profile

The task has 50 queries and 5,035 documents. It contains 57 positive qrels, with 1.14 positives per query on average. The positives-per-query distribution is 1 minimum, 1.00 median, and 2 maximum, and 14.0% of queries are multi-positive. Queries average 29.30 characters, while documents average 274.21 characters. The examples ask about a sports event location, a film's origin, the meaning of a landmark's location, a constitutional clause, and a song credit.

BM25 Evaluation Profile

The BM25 top-500 subset reaches nDCG@10 = 0.4301, hit@10 = 0.6200, and Recall@100 = 0.7895. BM25 can exploit names, titles, and distinctive terms, but short Korean questions often do not share enough surface wording with the answer passage. Evidence may be expressed through surrounding encyclopedia context rather than direct repetition of the query. BM25 is therefore useful as an entity-aware baseline but limited for answer semantics.

Dense Evaluation Profile

The dense harrier-oss-270m top-500 subset reaches nDCG@10 = 0.5805, hit@10 = 0.8400, and Recall@100 = 0.9298. Dense retrieval is clearly the strongest top-10 profile. It connects question intent to answer-bearing passages more effectively than exact term matching, especially when the passage expresses the answer through explanation, date, relation, or context. Dense retrieval also substantially improves candidate coverage over BM25.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses 100 candidates per query and reaches nDCG@10 = 0.5033, hit@10 = 0.7200, and Recall@100 = 0.9825, with no rank-101 safeguard rows. Hybrid retrieval has the best Recall@100, but its top-10 ranking is weaker than dense retrieval. This means the hybrid candidate pool is excellent for downstream reranking, while dense retrieval is the stronger first-stage order when only the top ranks are used directly.

Metric Interpretation for Model Researchers

This task cleanly separates answer-ranking quality from coverage. BM25 is the weakest because lexical overlap undercaptures question-answer relations. Dense retrieval is best for nDCG@10 and hit@10. Hybrid retrieval is best for Recall@100, making it valuable when a reranker follows candidate generation. Model improvements should be described as semantic answer ranking, candidate coverage, or both.

Query and Relevance Type Tendencies

Queries are short open-domain questions. Relevant passages are Wikipedia-style summaries that contain the answer in context. The retriever must match the relation expressed by the question, such as where an event took place, why a landmark is located somewhere, or who participated in a song, not just the headline entity.

Representative Failure Modes

BM25 can retrieve the right entity but the wrong answer passage. Dense retrieval can retrieve semantically related Wikipedia passages that omit the specific answer. Hybrid retrieval can recover more positives in the candidate set while still ranking lexical or semantic distractors above the evidence. For the few two-positive queries, retrieving only one evidence passage remains a coverage error.

Training Data That May Help

Useful training data includes non-overlapping open-domain QA retrieval, Wikipedia question-passage pairs, Korean answer retrieval, and multilingual QA evidence retrieval. Hard negatives should contain related entities without the answer. Training should exclude Natural Questions, BEIR, NanoBEIR, and overlapping translated passages from this benchmark.

Model Improvement Notes

Strong systems should represent short Korean questions as answer-seeking intents while preserving entity precision. Dense retrieval is a strong direct ranker; hybrid candidates are useful for reranking experiments that seek higher coverage without losing semantic answer ordering.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets