MNanoBEIR / NanoBEIR-en / NanoSciFact
Overview#
NanoSciFact is the compact English NanoBEIR version of SciFact, a scientific claim evidence retrieval task. Each query is a scientific claim, and the corpus contains research abstracts that can support or refute it. The retrieval goal is to surface evidence abstracts for downstream scientific verification. This makes the task useful for evaluating biomedical evidence retrieval, scientific terminology handling, and claim-to-abstract ranking.
Details#
What the Original Data Measures#
SciFact was designed for verifying scientific claims against the research literature. The full task includes selecting evidence abstracts, identifying rationale sentences, and determining whether the evidence supports or refutes the claim. In retrieval form, the first step is evaluated: can the system retrieve the abstracts that contain the evidence?
The NanoBEIR version keeps this scientific claim-to-evidence structure. A strong retriever must match a precise claim to an abstract that addresses it, even when the abstract uses technical wording, measurements, interventions, or outcomes rather than repeating the claim verbatim.
Observed Data Profile#
The task contains 50 queries, 2,919 documents, and 56 relevance judgments. Most queries have one positive abstract, with an average of 1.12 positives per query. The minimum is 1, the median is 1.0, the maximum is 4, and 4 queries are multi-positive, or 8.0% of the set.
Queries average 95.80 characters, while documents average 1,431.23 characters. Claims are longer and more technical than ordinary web questions, and documents are long scientific abstracts. The model must align a compact claim with the study evidence that can verify it.
BM25 Evaluation Profile#
BM25 reaches nDCG@10 of 0.7282, hit@10 of 0.8800, and recall@100 of 0.9464 using the top-500 BM25 candidate subset. This is a strong lexical baseline. Scientific claims often contain distinctive biomedical terms, abbreviations, proteins, diseases, methods, or interventions that appear in the relevant abstract.
The remaining failures occur when shared terminology is not enough to identify the verifying abstract. BM25 can retrieve abstracts about the same entity or process while missing the paper that actually supports or refutes the claim. Exact terms help, but evidence relation still matters.
Dense Evaluation Profile#
The dense harrier-oss-270m run reaches nDCG@10 of 0.7679, hit@10 of 0.8400, and recall@100 of 0.9286. Dense retrieval has the best nDCG@10, indicating stronger top-rank ordering when it finds the evidence. It is slightly weaker than BM25 on hit@10 and recall@100.
This pattern suggests that embedding similarity helps rank scientific evidence by claim meaning, but exact biomedical anchors remain important for full candidate coverage. Dense retrieval is useful for paraphrased or relation-oriented claims, while sparse matching protects rare technical names.
Reranking Hybrid Evaluation Profile#
The reranking_hybrid candidate set reaches nDCG@10 of 0.7397, hit@10 of 0.8800, and recall@100 of 0.9821. It uses a top-100 candidate range with an optional rank-101 safeguard; this slice has 1 safeguard row, candidate counts from 100 to 101, and a mean of 100.02 candidates. The hybrid profile has the best recall@100 and matches BM25 hit@10, while dense retrieval has the best nDCG@10.
This makes reranking_hybrid the strongest candidate pool for downstream scientific verification. It combines sparse technical-term coverage with dense semantic matching and gives a verifier the highest chance of seeing the evidence abstract within the top 100.
Metric Interpretation for Model Researchers#
Because most queries have one positive, hit@10 measures whether the evidence abstract is visible, and nDCG@10 measures how early it is ranked. recall@100 is critical for verification pipelines because later claim classification cannot recover missing evidence.
The comparison shows a useful split: BM25 is strong for terminology, dense retrieval is strongest for direct top-rank evidence ordering, and reranking_hybrid is best for candidate completeness. This task is a good diagnostic for balancing biomedical lexical precision with scientific semantic matching.
Query and Relevance Type Tendencies#
Queries include claims about Ly49Q and neutrophil migration, antiretroviral therapy and tuberculosis, interferon-induced genes and West Nile virus, HPV screening sensitivity, and TDP-43 interactions with respiratory complex proteins. Relevant documents are scientific abstracts with background, methods, results, and conclusions.
The task rewards evidence relation matching. A passage must address the claim's biological or clinical relation, not merely mention the same disease, protein, or method. Same-entity scientific abstracts are hard negatives.
Representative Failure Modes#
Likely failures include retrieving abstracts that share terminology but test a different hypothesis, missing evidence because the wording is paraphrased, confusing direction of effect, and over-ranking broad background abstracts. BM25 may be too term-driven, while dense retrieval may underweight rare biomedical identifiers.
Training Data That May Help#
Useful training data includes non-overlapping SciFact training claims, scientific claim and evidence abstract pairs, biomedical entailment data, scientific NLI, citation-sentence-to-cited-abstract supervision, and hard negatives that share terms but do not verify the claim. Both support and refute evidence should be represented.
Model Improvement Notes#
A model targeting this task should preserve exact biomedical identifiers while improving claim-evidence relation ranking. Sparse systems need strong biomedical tokenization and acronym handling. Dense systems need scientific claim-to-abstract training. Hybrid systems are promising for reranking because they provide the highest evidence coverage.
Example Data#
| Query | Positive document |
| Ly49Q directs the organization of neutrophil migration to inflammation sites by regulating membrane raft functions. [115 chars] | Neutrophils rapidly undergo polarization and directional movement to infiltrate the sites of infection and inflammation. Here, we show that an inhibitory MHC I receptor, Ly49Q, was crucial for the swift polarization of and tissue infiltration by neutrophils. During the steady state, Ly49Q inhibited neutrophil adhesion by preventing focal-complex formation, likely by inhibiting Src and PI3 kinases. However, in the presence of inflammatory stimuli, Ly49Q mediated rapid neutrophil polarization and tissue infiltration in an ITIM-domain-dependent manner. These opposite functions appeared to be mediated by distinct use of effector phosphatase SHP-1 and SHP-2. Ly49Q-dependent polarization and migration were affected by Ly49Q regulation of membrane raft functions. We propose that Ly49Q is pivotal in switching neutrophils to their polarized morphology and rapid migration upon inflammation, through its spatiotemporal regulation of membrane rafts and raft-associated signaling molecules. [990 chars] |
| Antiretroviral therapy reduces rates of tuberculosis across a broad range of CD4 strata. [88 chars] | BACKGROUND Human immunodeficiency virus (HIV) infection is the strongest risk factor for developing tuberculosis and has fuelled its resurgence, especially in sub-Saharan Africa. In 2010, there were an estimated 1.1 million incident cases of tuberculosis among the 34 million people living with HIV worldwide. Antiretroviral therapy has substantial potential to prevent HIV-associated tuberculosis. We conducted a systematic review of studies that analysed the impact of antiretroviral therapy on the incidence of tuberculosis in adults with HIV infection. METHODS AND FINDINGS PubMed, Embase, African Index Medicus, LILACS, and clinical trial registries were systematically searched. Randomised controlled trials, prospective cohort studies, and retrospective cohort studies were included if they compared tuberculosis incidence by antiretroviral therapy status in HIV-infected adults for a median of over 6 mo in developing countries. For the meta-analyses there were four categories based on CD4 c... [1,000 / 2,144 chars] |
| Rapid up-regulation and higher basal expression of interferon-induced genes reduce survival of granule cell neurons that are infected by West Nile virus. [153 chars] | Although susceptibility of neurons in the brain to microbial infection is a major determinant of clinical outcome, little is known about the molecular factors governing this vulnerability. Here we show that two types of neurons from distinct brain regions showed differential permissivity to replication of several positive-stranded RNA viruses. Granule cell neurons of the cerebellum and cortical neurons from the cerebral cortex have unique innate immune programs that confer differential susceptibility to viral infection ex vivo and in vivo. By transducing cortical neurons with genes that were expressed more highly in granule cell neurons, we identified three interferon-stimulated genes (ISGs; Ifi27, Irg1 and Rsad2 (also known as Viperin)) that mediated the antiviral effects against different neurotropic viruses. Moreover, we found that the epigenetic state and microRNA (miRNA)-mediated regulation of ISGs correlates with enhanced antiviral response in granule cell neurons. Thus, neurons... [1,000 / 1,153 chars] |
Source Reference Table#
| Item | Reference |
| Original dataset paper | SciFact |
| Proceedings page | SciFact EMNLP page |
| Project repository | SciFact GitHub repository |
| Retrieval benchmark framing | BEIR |
| Source dataset card | mteb/scifact |
Representative query and positive evidence snippets:
| Query | Positive document snippet |
| Ly49Q directs the organization of neutrophil migration to inflammation sites by regulating membrane raft functions. | Neutrophils rapidly undergo polarization and directional movement to infiltrate infection and inflammation sites. |
| Antiretroviral therapy reduces rates of tuberculosis across a broad range of CD4 strata. | HIV infection is the strongest risk factor for developing tuberculosis and has fueled its resurgence. |
| Rapid up-regulation and higher basal expression of interferon-induced genes reduce survival of granule cell neurons infected by West Nile virus. | The abstract discusses molecular factors governing neuronal susceptibility to microbial infection. |
| Primary cervical cancer screening with HPV detection has higher longitudinal sensitivity than conventional cytology. | HPV testing increases sensitivity for detecting high-grade cervical intraepithelial neoplasia. |
| Blocking the interaction between TDP-43 and respiratory complex I proteins ND3 and ND6 leads to increased neuronal loss. | Mutations in TAR DNA-binding protein 43 are connected to amyotrophic lateral sclerosis. |
Dataset Information#
| Field | Value |
| Nano set | MNanoBEIR |
| Backing dataset | NanoBEIR-en |
| Task / split | NanoSciFact |
| Hugging Face dataset | hakari-bench/NanoBEIR-en |
| Language | en |
| Category | natural_language |
| Queries | 50 |
| Documents | 2,919 |
| Positive qrels | 56 |
| Positives / query avg | 1.12 |
| Positives / query min | 1 |
| Positives / query median | 1.00 |
| Positives / query max | 4 |
| Multi-positive queries | 4 (8.00%) |
| Query length avg chars | 95.80 |
| Document length avg chars | 1,431.23 |
Candidate Subsets#
| Profile | Config | nDCG@10 | Hit@10 | Recall@100 | Candidates |
| BM25 | bm25 | 0.7282 | 0.8800 | 0.9464 | top-500 |
| Dense | harrier_oss_v1_270m | 0.7679 | 0.8400 | 0.9286 | top-500 |
| Reranking hybrid | reranking_hybrid | 0.7397 | 0.8800 | 0.9821 | top-100 |
Training and Leakage Metadata#
- Original train split: available
- Evaluation split origin: unknown
- Train/eval overlap audit: not_audited
- Leakage note: prefer excluding upstream SciFact dev/test claims, evidence abstracts, rationales, and Nano evaluation qrels from training
- Multi-positive training: single_positive_question_document_focus
- Useful training data: official non-overlapping SciFact train split, scientific claim and evidence abstract pairs, biomedical entailment and scientific NLI data, citation-sentence to cited-abstract supervision