NanoMuPLeR / el

Overview

NanoMuPLeR / el is the Greek split of MuPLeR-retrieval, a multilingual legal retrieval benchmark built from European Union legal passages. Queries are synthetic Greek legal questions, and documents are Greek passages aligned with DGT-Acquis material. Each query has one relevant passage. The task is useful for evaluating same-language Greek legal retrieval where formal EU terminology, institutional names, numeric thresholds, article references, and legal conditions must be matched precisely. It also supports comparison with other MuPLeR language splits because the underlying legal material and query design are parallel across languages.

Details

What the Original Data Measures

MuPLeR-retrieval measures multilingual parallel legal retrieval over DGT-Acquis passages. The source dataset card describes 10,000 human-translated EU legal passages and 200 synthetic parallel queries per language. The DGT-Acquis source belongs to the European Union's multilingual legal corpus resources.

For this Greek split, both query and document text are Greek. The retrieval target is the passage that grounds the legal condition, institution, threshold, date, or procedural rule asked about in the query.

Observed Data Profile

The Nano split contains 200 queries, 10,000 documents, and 200 positive qrel rows. Each query has exactly one positive. Queries average 141.28 characters, while documents average 744.82 characters.

The examples include questions about social clauses, state aid, EU budget control, data retention for maintenance claims, and research-program funding measures. Documents are formal legal or administrative passages with dense institutional wording.

BM25 Evaluation Profile

The BM25 candidate subset uses top-500 candidates and reaches nDCG@10 of 0.7749, hit@10 of 0.8600, and recall@100 of 0.9500. BM25 is strong because synthetic legal questions often preserve exact legal terms, institutions, dates, percentages, and named bodies from the positive passage.

Its weaknesses are legal paraphrase and condition matching. A wrong passage can share the same institution or policy area but not answer the exact legal condition in the query.

Dense Evaluation Profile

The dense candidate subset from harrier_oss_v1_270m uses top-500 candidates and reaches nDCG@10 of 0.7834, hit@10 of 0.8650, and recall@100 of 0.9450. Dense retrieval is slightly stronger than BM25 in top-rank quality, while BM25 is slightly stronger in recall@100.

This indicates that Greek semantic matching adds value over exact term matching, but sparse legal anchors remain important. The task rewards models that can preserve formal Greek legal meaning while still recognizing exact references.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses top-100 candidates, with one query carrying a rank-101 safeguard positive. It reaches nDCG@10 of 0.8390, hit@10 of 0.9150, and recall@100 of 0.9950. This is the strongest profile across the candidate types.

Hybrid retrieval is therefore the best candidate-generation strategy for this split. It combines BM25's exact legal terminology with dense retrieval's ability to match paraphrased legal conditions.

Metric Interpretation for Model Researchers

This is a single-positive task, so nDCG@10 and hit@10 directly reflect whether the exact grounding passage is ranked early. Recall@100 measures whether the correct legal passage is available to a reranker.

The strong hybrid result suggests that Greek legal retrieval should be evaluated with both sparse and dense signals, especially when queries contain exact legal anchors but also synthetic paraphrase.

Query and Relevance Type Tendencies

Queries are formal Greek legal questions. Relevant documents are Greek EU legal passages. The questions often ask which institution, provision, committee, period, or rule satisfies a specific condition.

The relevance relation is exact legal grounding. A passage about the same EU topic is not enough unless it answers the specific condition.

Representative Failure Modes

Common failures include retrieving a nearby EU provision, matching an institution name without the required action, confusing similar policy programs, and over-weighting repeated legal formulae. Dense systems may blur legal distinctions; sparse systems may miss paraphrased conditions.

Training Data That May Help

Useful training data includes non-overlapping Greek EU legal retrieval pairs, DGT-Acquis or EUR-Lex passages, multilingual legal QA, and hard negatives from adjacent EU acts. MuPLeR evaluation queries and exact positive passages should be excluded.

Model Improvement Notes

Models should preserve Greek morphology, legal terminology, article references, and institutional names. Hard negatives should share the same legal domain but fail the query's exact condition. Hybrid retrieval and reranking are especially appropriate because the task benefits from both exact and semantic signals.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets