MNanoBEIR / NanoBEIR-ko / NanoSciFact

Overview

NanoBEIR-ko__NanoSciFact is the Korean NanoBEIR version of SciFact, a scientific claim verification retrieval benchmark. The task uses Korean translated scientific claims as queries and asks a retriever to rank Korean translated abstracts that provide evidence for support or refutation. The Nano split contains 50 queries, 2,919 documents, and 56 positive qrels. Most queries have one positive, while 4 queries have multiple positives. This is a claim-to-evidence task where exact biomedical terminology is highly predictive, but semantic context still matters for ranking the strongest evidence.

Details

What the Original Data Measures

Fact or Fiction: Verifying Scientific Claims introduced SciFact as expert-written scientific claims paired with evidence abstracts, support/refute labels, and rationales. BEIR evaluates the retrieval step: a verifier cannot decide a claim unless the evidence abstract is first retrieved. In this Korean NanoBEIR version, translated biomedical claims are matched against translated abstracts, testing whether a model can preserve scientific entity specificity while matching the claim's finding.

Observed Data Profile

The task has 50 queries and 2,919 documents. It contains 56 positive qrels, with 1.12 positives per query on average. The positives-per-query distribution is 1 minimum, 1.00 median, and 4 maximum, and only 8.0% of queries are multi-positive. Queries average 46.28 characters, while documents average 723.55 characters. The examples cover neutrophil migration, antiretroviral therapy and tuberculosis, interferon-induced genes, cervical cancer screening, and TDP-43 interactions.

BM25 Evaluation Profile

The BM25 top-500 subset reaches nDCG@10 = 0.6835, hit@10 = 0.8600, and Recall@100 = 0.9286. BM25 is very strong because claims and abstracts often share distinctive genes, proteins, diseases, interventions, and outcome terms. Exact term matching gives a reliable candidate pool and the best coverage. Its remaining limitation is rank ordering: a term-overlapping abstract can discuss the same entity without bearing on the exact claim.

Dense Evaluation Profile

The dense harrier-oss-270m top-500 subset reaches nDCG@10 = 0.6207, hit@10 = 0.7800, and Recall@100 = 0.8571. Dense retrieval is weaker than BM25 on this split. The result suggests that general semantic similarity can blur fine biomedical distinctions. A passage about the same molecule, disease, or therapy may be semantically close but not evidence for the exact claim. Dense retrieval therefore needs domain-specific hard-negative training to compete here.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses 100 to 101 candidates per query and reaches nDCG@10 = 0.6838, hit@10 = 0.8400, and Recall@100 = 0.9107. Five queries use the rank-101 safeguard. Hybrid retrieval is very slightly strongest on nDCG@10, while BM25 remains stronger on hit@10 and Recall@100. This means that combining lexical and dense signals can improve top-rank ordering, but lexical candidate coverage remains essential for Korean SciFact.

Metric Interpretation for Model Researchers

This task is not dense-dominant. BM25 is a hard baseline because biomedical claim retrieval depends on exact terminology and relation specificity. reranking_hybrid improves nDCG@10 only marginally, suggesting that dense signals help order evidence when they reinforce lexical matches. Researchers should preserve entity, intervention, and outcome precision rather than relying on broad semantic similarity.

Query and Relevance Type Tendencies

Queries are concise scientific claims. Relevant documents are abstracts that describe experiments, observations, or clinical findings related to those claims. Relevance depends on the exact scientific relation. A paper that shares TDP-43, HPV, CD4, or interferon terminology can still be a distractor if it does not support or refute the claim.

Representative Failure Modes

BM25 can over-rank abstracts that share the right terms but describe a different finding. Dense retrieval can over-rank same-domain abstracts that are too broad or scientifically adjacent. Hybrid retrieval can improve top-rank ordering but still lose some BM25 coverage. Retrieval errors are especially harmful because downstream verification cannot recover if the evidence abstract is absent.

Training Data That May Help

Useful training data includes non-overlapping scientific fact verification, claim-evidence retrieval, biomedical abstract retrieval, and Korean or multilingual scientific NLI. Hard negatives should share terminology with the claim but differ in the finding, intervention, or outcome. Training should exclude SciFact, BEIR, NanoBEIR, and overlapping translated abstracts from this benchmark.

Model Improvement Notes

Strong systems should combine exact biomedical lexical recall with claim-level semantic matching. Rerankers should attend to relation direction, experimental context, and whether the abstract actually bears on the claim. For this task, preserving BM25-like candidate coverage is a practical requirement.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets