MNanoBEIR / NanoBEIR-th / NanoSciFact

Overview

NanoSciFact in the Thai NanoBEIR slice is a scientific evidence retrieval task derived from SciFact. The queries are Thai translated scientific claims, and the corpus contains Thai translated scientific abstracts. The retrieval goal is to find abstracts that provide evidence for each claim, not merely papers in the same broad topic area. This makes the task a compact benchmark for Thai scientific claim retrieval, biomedical evidence matching, and fact-checking-oriented search.

Details

What the Original Data Measures

SciFact evaluates scientific claim verification using research abstracts. In retrieval form, a model receives a claim and must retrieve an abstract containing evidence needed to assess it. The relevant abstract may discuss a specific biological mechanism, clinical result, intervention, gene, disease, or experimental finding.

The Thai translated version adds multilingual and domain-specific pressure. Claims are long and technical, and abstracts contain dense biomedical language. A strong retriever must preserve exact scientific terms such as gene names or diseases while also recognizing the evidence relation when the abstract phrases it differently.

Observed Data Profile

The task contains 50 queries, 2,919 documents, and 56 relevance judgments. Most queries have one positive abstract, with an average of 1.12 positives per query. The minimum is 1, the median is 1.0, the maximum is 4, and 4 queries are multi-positive, or 8.0% of the query set. This is mostly a single-evidence retrieval task.

Queries average 92.74 characters, while documents average 1,328.85 characters. The claims are already fairly long, but the abstracts are much longer and may include background, method, result, and conclusion text. The model must identify the abstract that evidences the claim, not only the abstract that shares the most terminology.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.6334, hit@10 of 0.7800, and recall@100 of 0.8571 using the top-500 BM25 candidate subset. This is a strong lexical baseline. Scientific claims often contain distinctive biomedical terms, and exact matching can identify many candidate abstracts.

The result still leaves room for evidence-sensitive ranking. A same-entity abstract may not support the claim, and a relevant abstract may express the same finding through different wording. BM25 can locate the right scientific neighborhood, but it does not fully model the claim-evidence relation.

Dense Evaluation Profile

The dense harrier-oss-270m run reaches nDCG@10 of 0.5713, hit@10 of 0.7200, and recall@100 of 0.8571. Dense retrieval matches BM25 recall@100 but is weaker on top-rank metrics. This suggests that exact scientific terminology is especially important in this Thai SciFact slice, and the dense model may over-rank semantically related but non-evidential abstracts.

The dense profile remains useful as a semantic complement, but it does not outperform lexical matching directly. Scientific evidence retrieval requires precise relation matching, not only broad semantic relatedness. General embeddings can blur abstracts that discuss the same disease, gene, or method while differing in the claim outcome.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate set reaches nDCG@10 of 0.6206, hit@10 of 0.7400, and recall@100 of 0.9286. It uses a top-100 candidate range with an optional rank-101 safeguard; the observed candidate count ranges from 100 to 101, with 4 safeguard rows and a mean of 100.08 candidates. The hybrid profile has the best recall@100, while BM25 has the strongest nDCG@10 and hit@10.

This indicates that hybrid search improves evidence candidate coverage but does not fully improve first-page ordering. Combining lexical and dense candidates helps recover more positives by rank 100, but the top ranks still depend heavily on precise biomedical term matching. A downstream reranker could benefit from the hybrid pool if it can judge the claim-abstract evidence relation.

Metric Interpretation for Model Researchers

nDCG@10 is the key direct ranking metric because scientific evidence should appear in the first page for a verifier or reader. hit@10 measures whether at least one evidence abstract is visible, while recall@100 measures whether a reranker has access to the evidence. Since most queries have one positive, top-rank ordering is especially important.

The method comparison shows that Thai SciFact is lexically anchored. BM25 performs best at the top, dense retrieval does not improve ordering, and reranking_hybrid improves recall. This is useful for diagnosing whether a model can handle precise scientific terminology and whether hybrid search provides candidate coverage without degrading top-rank evidence quality.

Query and Relevance Type Tendencies

Queries are long scientific claims, often biomedical. Examples involve Ly49Q and neutrophil migration, antiretroviral therapy and tuberculosis, interferon-induced genes and West Nile virus, HPV detection for cervical cancer screening, and TDP-43 interactions in neuronal loss. Relevant documents are abstracts containing experimental or clinical evidence.

The task rewards models that preserve technical detail. Small changes in genes, proteins, diseases, treatments, or causal relations can change relevance. A broadly related biomedical abstract is not enough if it does not provide the evidence needed for the claim.

Representative Failure Modes

Likely failures include retrieving abstracts that mention the same entity but not the claim outcome, confusing support evidence with background, missing evidence because Thai translation changes terminology, and over-ranking abstracts with similar biomedical vocabulary. Dense models may retrieve same-topic non-evidence, while BM25 may miss paraphrased evidence.

Training Data That May Help

Useful training data includes scientific claim verification, biomedical abstract retrieval, Thai scientific QA, multilingual evidence retrieval, and hard negatives that share entities or methods but do not support the claim. Thai biomedical terminology resources and segmentation-aware preprocessing can help. For rerankers, close non-evidence abstracts are especially valuable.

Model Improvement Notes

A model targeting this task should combine biomedical term precision with evidence-aware reasoning. Sparse systems should maintain strong exact matching for scientific names. Dense systems need domain-specific hard-negative training to distinguish evidence from topical relatedness. Hybrid systems should exploit their stronger recall with a reranker that compares the claim and abstract in detail.

Example Data

Source Reference Table

Representative query and positive abstract snippets:

Dataset Information

Candidate Subsets