MNanoBEIR / NanoBEIR-ko / NanoArguAna

Overview

NanoBEIR-ko__NanoArguAna is the Korean NanoBEIR version of ArguAna, an argument and counterargument retrieval benchmark. The task uses Korean translated argumentative passages as queries and asks a retriever to find the paired Korean translated counterargument or closely matched argumentative response. The Nano split contains 50 queries, 3,635 documents, and 50 positive qrels, with exactly one positive document per query. Queries and documents are both long compared with ordinary search tasks, so the benchmark tests whether a model can match stance, premise, and argumentative relation rather than only topic words.

Details

What the Original Data Measures

ArguAna was introduced for argument retrieval and matching in debate-style text. BEIR includes ArguAna as an argument retrieval task where the relevant document is not a factual answer but an argumentative counterpart. In this Korean NanoBEIR version, long translated claims and supporting passages are used as queries, and the positive document usually responds to the same issue with a counterargument or closely paired argument. This makes relevance discourse-oriented and stance-aware.

Observed Data Profile

The task has 50 queries and 3,635 documents. It contains 50 positive qrels, and every query has exactly one positive. Queries average 619.40 characters, while documents average 519.64 characters. The examples discuss public indifference to reform, Heathrow expansion, choice overload, cyber attacks, and religious speech. These long passages include several claims and supporting details, which gives lexical retrieval many anchors but also creates many same-topic distractors.

BM25 Evaluation Profile

The BM25 top-500 subset reaches nDCG@10 = 0.3661, hit@10 = 0.6600, and Recall@100 = 0.9000. BM25 is reasonably strong for candidate recall because long argumentative passages repeat topic terms and policy vocabulary. However, its top-10 ranking is limited. The correct paired counterargument is often not the passage with the highest surface overlap; it is the one that responds to the same premise or challenges the same claim. Topic-level lexical matching therefore leaves many hard negatives near the top.

Dense Evaluation Profile

The dense harrier-oss-270m top-500 subset reaches nDCG@10 = 0.4082, hit@10 = 0.6800, and Recall@100 = 0.9400. Dense retrieval improves over BM25 on all reported metrics, which indicates that semantic representations help connect argumentative passages by meaning and response relation. Dense retrieval is better at capturing paraphrased premises and stance-level similarity, but it still does not fully solve the single-positive ranking problem.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses 100 to 101 candidates per query and reaches nDCG@10 = 0.4217, hit@10 = 0.7800, and Recall@100 = 0.9600. Two queries use the rank-101 safeguard. Hybrid retrieval is the strongest profile across all three metrics. This shows that lexical and dense signals are complementary for Korean ArguAna: BM25 preserves topic and phrase anchors, while dense retrieval adds semantic and argumentative relatedness. The combined candidate set is both broader and better ranked than either single retrieval family.

Metric Interpretation for Model Researchers

This task is a clear hybrid-strength case. BM25 alone has good top-100 recall but weak early precision, dense retrieval improves semantic ranking, and reranking_hybrid gives the best top-10 and top-100 behavior. Researchers should interpret improvements as evidence of argument-level matching, not just long-text similarity. Because every query has one positive, a model must separate the exact paired response from many same-topic argument passages.

Query and Relevance Type Tendencies

The examples are long debate passages where the positive document challenges, qualifies, or reframes the query's claim. The relationship can involve counterevidence, a distinction between public opinion and misinformation, an alternative economic analysis, or a legal principle. Relevance depends on responding to the same argument structure, not merely discussing the same topic.

Representative Failure Modes

BM25 can retrieve passages that share issue words but respond to a different premise. Dense retrieval can retrieve semantically related debate text that is not the paired counterargument. Hybrid retrieval can still fail when both signals favor broad same-topic documents. Long Korean translated passages also increase the chance that one paragraph overlaps strongly while the overall argument relation is wrong.

Training Data That May Help

Useful training data includes non-overlapping argument retrieval, debate counterargument pairs, stance-aware retrieval, Korean argument mining, and multilingual debate-pair data. Hard negatives should address the same topic while taking a different stance or responding to a different premise. Training should exclude ArguAna, BEIR, NanoBEIR, and translated argument records likely to overlap with this benchmark.

Model Improvement Notes

Strong systems should encode long argument passages without reducing them to topic vectors. Useful directions include stance-aware contrastive learning, premise-response hard negatives, and reranking signals that compare the main claim, evidence, and counterclaim. Hybrid candidate generation is especially appropriate because both lexical anchors and semantic argument structure matter.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets