NanoMTEB-Korean / miracl_ko

Overview

miracl_ko is the Korean MIRACL retrieval task. Queries are Korean information needs, and documents are Korean Wikipedia passages. The Nano split contains 200 queries, 10,000 documents, and 508 positive qrels. It is multi-positive: each query has 2.54 positives on average, the median is two, and 51.5% of queries have more than one relevant passage. Queries average 21.71 characters, while documents average 193.24 characters. The task tests Korean passage retrieval over short encyclopedic passages with native-language queries, aliases, morphology, and multiple answer-bearing contexts.

Details

What the Original Data Measures

MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages introduced a multilingual retrieval benchmark with human-annotated passage-level relevance judgments over Wikipedia. MIRACL includes Korean and provides native-language queries, passage corpora, and relevance annotations. The dataset was designed to improve multilingual retrieval coverage beyond English-centric benchmarks.

The Korean split is a same-language Wikipedia retrieval task. The model must rank answer-bearing or contextually relevant Korean passages for a Korean query, often with more than one positive passage.

Observed Data Profile

The Nano split has 200 Korean queries, 10,000 documents, and 508 positive judgments. Documents are short Korean Wikipedia passages with title prefixes. Queries are compact fact-oriented questions, often asking about people, places, religion, science, history, and definitions.

Examples include questions about Hercules, King Sukjong, Catholic canon law, RNA structure, and the origin of Buddhism. Passage segmentation means several passages can be relevant to one query, and some positives may provide supporting context rather than a single direct answer sentence.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.5069, hit@10 of 0.8050, and recall@100 of 0.9606. This is a strong lexical baseline for Korean Wikipedia retrieval. Named entities, titles, and direct fact terms often provide enough overlap for BM25 to retrieve relevant passages.

BM25 is not the best top-rank method, however. Korean morphology, aliases, synonyms, and short query length can make exact term matching fragile. BM25 has high recall but can place the most useful passage below semantically related distractors.

Dense Evaluation Profile

Dense retrieval is stronger at early ranking, with nDCG@10 of 0.6997, hit@10 of 0.9150, and recall@100 of 0.9291. Dense retrieval better captures semantic relationships between compact Korean questions and answer-bearing passages. It can handle paraphrase and entity context more effectively than pure lexical matching.

Dense recall@100 is lower than BM25, which shows the tradeoff: dense retrieval often ranks better when it finds the right evidence, but may miss some lexical positives in the first 100. This is important for reranking pipeline design.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile is strongest overall, with nDCG@10 of 0.7121, hit@10 of 0.9550, and recall@100 of 0.9902. It combines the dense model's semantic ranking advantage with BM25's lexical coverage. Candidate lists contain 100 to 101 entries, with three safeguard-positive rows.

This is the ideal hybrid pattern: hybrid search improves both early ranking and candidate coverage relative to the individual methods. It is a strong candidate source for downstream reranking in Korean Wikipedia retrieval.

Metric Interpretation for Model Researchers

miracl_ko is hybrid-favorable, with dense retrieval strong at top-10 and BM25 strong for recall. The task is useful for evaluating whether a model can balance Korean lexical anchors with semantic matching. Since many queries have multiple positives, recall@100 should be interpreted as coverage across all useful evidence passages, not only one answer.

nDCG@10 measures whether the most useful relevant passages appear early. Hit@10 measures whether at least one relevant passage is retrieved for a user or QA system. The hybrid score indicates that combining lexical and dense evidence is especially valuable here.

Query and Relevance Type Tendencies

Queries are short Korean information needs, often asking who, what, where, or whether a statement is true. Positive documents are Korean Wikipedia passages with answer evidence or necessary background. Topics range from mythology and history to biology, religion, politics, and geography.

Relevance can be multi-passage. A query may have several passages that answer or support the information need, so training and evaluation should allow multiple positives.

Representative Failure Modes

BM25 can fail on aliases, inflection, and paraphrase. Dense retrieval can miss rare names or exact Korean title matches when semantic similarity is too broad. Hybrid retrieval reduces both risks, but can still rank a related passage above the most directly answer-bearing one.

Passage segmentation is another challenge: a passage may contain context but not the answer sentence, while another passage from the same article gives the direct answer.

Training Data That May Help

Useful training data includes non-overlapping MIRACL Korean train pairs, Mr. TyDi Korean retrieval pairs, Korean Wikipedia question-to-passage retrieval pairs, and native Korean hard negatives from the same Wikipedia topic. Training should exclude MIRACL Korean dev/test queries, qrels, and positive passages likely to overlap with the Nano split.

Synthetic data should create Korean Wikipedia-style passages with titles, aliases, dates, places, organizations, and definitions, then generate Korean fact questions grounded in one or more source passages. Multi-positive training is appropriate.

Model Improvement Notes

Models should combine Korean lexical fidelity with semantic passage matching. Dense encoders need stronger handling of aliases, title variants, and rare entities. Hybrid and reranking systems should preserve multiple relevant passages while prioritizing the most answer-bearing one.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: dev
• Train/eval overlap audit: not_audited
• Leakage note: exclude MIRACL Korean dev/test queries, qrels, and positive passages likely to overlap with the Nano split
• Multi-positive training: multi_positive_objective
• Useful training data: non-overlapping MIRACL Korean train pairs, Mr. TyDi Korean retrieval pairs, Korean Wikipedia question-to-passage retrieval pairs, native Korean hard negatives from the same Wikipedia topic