NanoMuPLeR / fi

Overview

NanoMuPLeR / fi is the Finnish split of MuPLeR-retrieval. It uses synthetic Finnish legal questions and Finnish DGT-Acquis-derived EU legal passages. Each query has one positive passage that contains the legal condition or factual detail required by the question. The split is useful for evaluating retrieval in a morphologically rich language where long Finnish compounds, case inflection, EU institutional terms, and legal formulae all affect matching. Compared with other MuPLeR languages, Finnish has especially long queries in this Nano sample, making it a good test of robust legal term and condition matching.

Details

What the Original Data Measures

MuPLeR-retrieval measures multilingual parallel legal retrieval using DGT-Acquis passages and synthetic queries. The source dataset card describes 10,000 human-translated EU legal passages and 200 synthetic queries per language.

For the Finnish split, retrieval is same-language. The model must find the passage that answers the legal question, not merely a passage in the same EU policy area.

Observed Data Profile

The Nano split contains 200 queries, 10,000 documents, and 200 positive qrel rows. Each query has exactly one positive. Queries average 160.16 characters, while documents average 683.64 characters.

Examples include maritime labor conventions, telecommunications market dominance, vocational training indicators, human-resource planning, and intelligent transport systems. Documents are formal Finnish EU legal or administrative passages.

BM25 Evaluation Profile

The BM25 candidate subset uses top-500 candidates and reaches nDCG@10 of 0.8230, hit@10 of 0.9050, and recall@100 of 0.9400. BM25 is very strong despite Finnish morphology because many queries contain distinctive legal terms, dates, institutions, and topic-specific vocabulary.

Its recall@100 is lower than dense retrieval's, suggesting that exact term overlap sometimes misses the positive when Finnish inflection or paraphrase changes surface forms.

Dense Evaluation Profile

The dense candidate subset from harrier_oss_v1_270m uses top-500 candidates and reaches nDCG@10 of 0.7955, hit@10 of 0.8850, and recall@100 of 0.9600. Dense retrieval has better recall@100 than BM25 but weaker top-rank quality.

This pattern suggests that dense retrieval broadens candidate coverage under Finnish variation, while BM25's exact legal anchors often place the right passage higher when it finds it.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses top-100 candidates, with four queries carrying a rank-101 safeguard positive. It reaches nDCG@10 of 0.8682, hit@10 of 0.9250, and recall@100 of 0.9800. This is the strongest profile across the candidate types.

Hybrid retrieval is therefore well suited for Finnish legal retrieval because it combines exact EU terminology with semantic matching across inflection and paraphrase.

Metric Interpretation for Model Researchers

This is single-positive retrieval, so top-rank metrics directly indicate exact passage grounding. Recall@100 shows candidate coverage for reranking. The split is especially useful for comparing sparse and dense behavior in a morphologically rich legal language.

BM25's high top-rank quality and dense retrieval's higher recall suggest that reranking over hybrid candidates is the most informative evaluation setup.

Query and Relevance Type Tendencies

Queries are long Finnish legal questions. Relevant documents are Finnish EU legal passages. The questions often ask which body, framework, indicator, plan, or system satisfies a specified condition.

The relevance relation is exact legal answerability. Same-domain legal text is insufficient if it does not satisfy the query condition.

Representative Failure Modes

Common failures include missing inflected variants, matching legal terms without the correct condition, confusing similar EU advisory or regulatory bodies, and over-ranking passages with shared policy vocabulary. Dense systems may retrieve broad legal-semantic matches; sparse systems may be brittle to morphology.

Training Data That May Help

Useful training data includes non-overlapping Finnish EU legal retrieval pairs, Finnish legal QA, EUR-Lex passages, multilingual parallel legal data, and hard negatives from nearby legal provisions. MuPLeR evaluation query-passage pairs and near-duplicate aligned passages should be excluded.

Model Improvement Notes

Models should handle Finnish morphology, compounds, legal terminology, and exact institutional references. Hybrid retrieval is particularly attractive because it balances exact matching with semantic coverage. Hard negatives should be legally adjacent but fail the specific condition in the query.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets