MNanoBEIR / NanoBEIR-ja / NanoSciFact

Overview

NanoBEIR-ja__NanoSciFact is the Japanese NanoBEIR version of SciFact, a scientific claim verification retrieval benchmark. The task uses Japanese translated scientific claims as queries and asks a retriever to rank Japanese translated abstracts that provide evidence for support or refutation. The Nano split contains 50 queries, 2,919 documents, and 56 positive qrels. Most queries have one positive, while 4 queries have multiple positives. This is a claim-to-evidence task rather than a related-paper task: the model must find the abstract whose experimental finding bears on a specific scientific claim.

Details

What the Original Data Measures

Fact or Fiction: Verifying Scientific Claims introduced SciFact as expert-written scientific claims paired with evidence abstracts, support/refute labels, and rationales. BEIR evaluates the retrieval component: before a verifier can decide whether a claim is supported or refuted, the system must retrieve the relevant abstract. In this Japanese NanoBEIR version, the evidence retrieval problem is tested through translated biomedical and scientific claims and translated abstracts.

Observed Data Profile

The task has 50 queries and 2,919 documents. It contains 56 positive qrels, with 1.12 positives per query on average. The positives-per-query distribution is 1 minimum, 1.00 median, and 4 maximum, and only 8.0% of queries are multi-positive. Queries average 40.58 characters, while documents average 633.08 characters. The examples cover neutrophil migration, antiretroviral therapy, interferon-induced genes, cervical cancer screening, and TDP-43 interactions. Claims are concise but terminology-heavy, and abstracts contain the experimental context needed for verification.

BM25 Evaluation Profile

The BM25 top-500 subset reaches nDCG@10 = 0.7023, hit@10 = 0.8600, and Recall@100 = 0.9464. BM25 is very strong because scientific claims often repeat distinctive entities, proteins, diseases, interventions, or outcome terms that also appear in the evidence abstract. Exact terminology is highly predictive in this task. The limitation is that evidence may use abbreviations, different phrasing, or broader experimental descriptions, so lexical matching alone does not always produce the best ranked evidence.

Dense Evaluation Profile

The dense harrier-oss-270m top-500 subset reaches nDCG@10 = 0.6751, hit@10 = 0.7800, and Recall@100 = 0.8750. Dense retrieval is competitive but below BM25 on all reported metrics. This suggests that general embedding similarity helps with scientific paraphrase, but it can lose the precise biomedical anchors that matter for claim verification. In SciFact, small entity and relation differences can change whether an abstract verifies a claim, so broad semantic relatedness is not enough.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses 100 to 101 candidates per query and reaches nDCG@10 = 0.7232, hit@10 = 0.8400, and Recall@100 = 0.9286. Four queries use the rank-101 safeguard. Hybrid retrieval has the best nDCG@10, while BM25 has the best hit@10 and Recall@100. This means lexical retrieval gives the broadest candidate coverage for this split, but hybrid fusion improves the ordering of the highest-ranked evidence when lexical and semantic signals reinforce each other.

Metric Interpretation for Model Researchers

This task is a strong reminder that scientific claim retrieval is not always dense-dominant. BM25 performs extremely well because exact biomedical terms are central to relevance. Dense retrieval alone can underperform when it blurs nearby scientific concepts. reranking_hybrid gives the best nDCG@10, so combining exact terminology with semantic evidence matching is valuable, but models should not discard lexical anchors. Improvements should preserve entity, intervention, and outcome specificity.

Query and Relevance Type Tendencies

Queries are atomic scientific claims. Relevant documents are abstracts that describe experiments, observations, or findings related to the claim. Many queries involve genes, proteins, viruses, therapies, or clinical endpoints. Relevance depends on the exact scientific relation, not merely on topic overlap. A passage about the same disease or molecule can be a hard negative if it does not address the claim's assertion.

Representative Failure Modes

BM25 can over-rank abstracts that repeat the right terms but describe a different finding. Dense retrieval can over-rank semantically related abstracts from the same biomedical area while missing the exact evidence. Hybrid retrieval can improve nDCG but still miss candidate coverage compared with BM25 when the semantic side adds noisy related documents. Retrieval errors are costly because a downstream verifier cannot support or refute a claim without the evidence abstract.

Training Data That May Help

Useful training data includes non-overlapping scientific fact verification, claim-evidence retrieval, biomedical abstract retrieval, and Japanese or multilingual scientific NLI. Hard negatives should share terminology with the claim but differ in the actual finding, intervention, or outcome. Training should exclude SciFact, BEIR, NanoBEIR, and overlapping translated abstracts from this benchmark.

Model Improvement Notes

Strong systems should combine exact biomedical term handling with claim-level semantic matching. Rerankers should attend to relation direction, experimental context, and whether the abstract actually bears on the claim. For this task, lexical recall is a high-value baseline, and dense models should be trained not to smooth away scientific specificity.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets