NanoMTEB-Korean / lawir_ko

Overview

lawir_ko is a Korean legal-information retrieval task based on statute and regulation articles. Queries ask for the legal provision associated with a Korean law title and article concept, and documents are individual article texts from Korean legal sources. The Nano split contains 200 queries, 3,562 documents, and 200 positive qrels, with exactly one positive article per query. Queries average 50.62 characters, and documents average 387.79 characters. The task is useful for evaluating whether Korean retrieval models can map a legal query to the correct article among many formulaic and cross-referenced provisions.

Details

What the Original Data Measures

No standalone task paper was confirmed for this dataset. The interpretation is based on the on-and-on/lawgov_ir-ko dataset card, Korea Law Information Center context, MTEB metadata, and observed Nano data. The dataset was created for Korean legal information retrieval, with documents representing statute or regulation articles.

In this retrieval formulation, the query often names a law and asks which provision covers a particular article concept. The model must identify the corresponding article text, not merely retrieve a legal document from the same topic area.

Observed Data Profile

The split has 200 Korean queries, 3,562 legal article documents, and 200 positive judgments. Every query has one positive. Queries usually mention a law title and a provision concept, while documents contain numbered clauses, definitions, duties, exceptions, and cross-references. The corpus includes many articles with similar formulaic language.

Examples involve financial consumer protection, food sanitation standards, agricultural product payment-settlement organizations, elderly-driver signs, and personal-information dispute procedures. The correct article is often distinguished by provision title and legal role rather than by broad domain.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.5232, hit@10 of 0.7100, and recall@100 of 0.9200. Lexical matching is helpful because law titles, article names, and specialized terms often appear in the query and article. However, BM25 is not sufficient: legal articles share common phrasing, enumerated structures, and cross-reference patterns, so high term overlap can point to adjacent but incorrect provisions.

BM25 failures are particularly important in legal retrieval because an adjacent article may look plausible but answer a different legal question. Exact article selection matters.

Dense Evaluation Profile

Dense retrieval is stronger, with nDCG@10 of 0.6534, hit@10 of 0.8200, and recall@100 of 0.9700. Dense embeddings appear to help connect the provision concept in the query to the relevant article even when the wording is not a perfect surface match. They also reduce some of the noise from boilerplate legal language.

This split is a good diagnostic for Korean legal-domain semantic matching. A model must preserve article names, legal concepts, and formal clause structure while avoiding generic topical similarity across the same law.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile reaches nDCG@10 of 0.6491, hit@10 of 0.8200, and recall@100 of 0.9750. It is nearly tied with dense retrieval at top-10 and slightly better for candidate coverage. Candidate lists contain 100 to 101 entries, with five safeguard-positive rows.

This is a balanced hybrid case. Dense retrieval gives strong semantic ranking, while lexical matching helps retain exact legal titles and provision terms. The hybrid pool is likely a strong input for a legal reranker because it offers the best top-100 relevant coverage.

Metric Interpretation for Model Researchers

lawir_ko is dense-favorable, with hybrid search best for recall. BM25 remains useful but is limited by legal boilerplate and adjacent provisions. Since each query has one positive, nDCG@10 and hit@10 directly evaluate whether the correct article is ranked early, while recall@100 indicates whether a downstream reranker can still recover it.

The dense-vs-BM25 gap suggests that Korean legal retrieval benefits from semantic article-title and provision-concept matching, not only exact term frequency.

Query and Relevance Type Tendencies

Queries are Korean legal retrieval requests naming a law and asking for the article that defines, applies, or explains a provision. Positive documents are single statute or regulation articles. They often contain numbered clauses, subclauses, and formal legal wording.

Relevance is provision-specific. Documents from the same law can be strong hard negatives if they share terms but govern a different article concept.

Representative Failure Modes

BM25 can over-rank adjacent articles with shared legal phrases. Dense retrieval can over-generalize within the same law or legal domain if it misses the exact article concept. Hybrid retrieval can improve coverage but still needs a reranker to resolve fine-grained provision distinctions.

Cross-references are another risk: a document may mention the right law or article number only as a reference, not as the governing provision requested by the query.

Training Data That May Help

Useful training data includes non-overlapping lawgov_ir-ko examples, Korean statute article retrieval pairs, law-title and article-title matching pairs, and hard negatives from the same law and adjacent provisions. Training should exclude lawgov_ir-ko evaluation rows, Nano queries, qrels, and positive statute articles.

Synthetic data should generate Korean statute articles with numbered clauses, article titles, definitions, duties, and cross-references, then create legal retrieval queries naming a law and asking for the article that explains a provision.

Model Improvement Notes

Models should preserve Korean legal terminology, article titles, and clause structure. Dense encoders need same-law hard negatives to avoid broad legal topic matching. Rerankers should check whether the candidate is the requested provision itself rather than an article that only references it.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: test
• Train/eval overlap audit: not_audited
• Leakage note: exclude lawgov_ir-ko evaluation rows, Nano queries, qrels, and positive statute articles
• Multi-positive training: single_positive_question_document_focus
• Useful training data: non-overlapping lawgov_ir-ko examples, Korean statute article retrieval pairs, law-title and article-title matching pairs, hard negatives from the same law and adjacent provisions