NanoChemTEB / NanoChemHotpotQA

Overview

NanoChemHotpotQA is the chemistry-filtered HotpotQA retrieval task from NanoChemTEB. Queries are English multi-hop questions whose answers are supported by Wikipedia passages connected to chemistry or chemistry-adjacent scientific topics. The task measures whether a retriever can follow bridge entities and scientific clues rather than simply matching one prominent term.

Details

What the Original Data Measures

ChemTEB includes ChemHotpotQARetrieval as a chemistry-domain retrieval task derived from HotpotQA. HotpotQA was designed for multi-hop question answering over Wikipedia, where questions often require connecting entities or properties across supporting facts.

The chemistry filtering makes the corpus scientific, but it does not remove the multi-hop character. Queries may ask about a scientist, organism, preservation method, physical measurement, or chemistry-adjacent entity through an indirect clue. The relevant passage contains the final answer fact, while hard negatives can share only the bridge term.

Observed Data Profile

The task contains 18 queries, 10,000 documents, and 18 relevance judgments. Every query has exactly one positive. Queries average 104.22 characters, and documents average 402.40 characters.

This is a very small Nano split, so metric differences are sensitive to a few examples. The documents are Wikipedia-style passages about scientists, fungi, food preservation, physics, plumbing, journals, and other scientific or chemistry-adjacent topics.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.7178, hit@10 of 0.7778, and recall@100 of 0.8889 using the top-500 BM25 candidate subset. This is a strong lexical baseline because many queries contain rare scientific entities or phrases that appear in the positive passage.

The remaining failures are typical multi-hop failures. A passage may share a scientific term or bridge entity while not containing the final answer. BM25 can retrieve the wrong scientist, fungus, or related scientific page when the query requires resolving the property being asked.

Dense Evaluation Profile

The dense harrier-oss-270m run reaches nDCG@10 of 0.7748, hit@10 of 0.8333, and recall@100 of 1.0000. Dense retrieval improves over BM25 and recovers every positive within the top 100.

This suggests that semantic matching helps connect the full multi-hop question to the answer-bearing passage. Dense retrieval is useful when the exact bridge term alone is not enough to select the correct supporting paragraph.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate set reaches nDCG@10 of 0.7923, hit@10 of 0.8333, and recall@100 of 1.0000. It uses exactly 100 candidates per query in this slice, with no rank-101 safeguard rows.

The hybrid profile is the strongest nDCG@10 result and ties dense retrieval for hit@10 and recall@100. It benefits from both rare entity overlap and semantic question-passage matching. Because the split is only 18 queries, the dense and hybrid difference should be read cautiously.

Metric Interpretation for Model Researchers

This task is a small chemistry-filtered multi-hop retrieval probe. BM25 is strong but not enough; dense and hybrid retrieval better handle the full question intent. The important distinction is whether the retriever finds the final answer passage, not merely a bridge-related page.

Researchers should treat the absolute scores with caution because of the small query count. The task is still useful as a diagnostic for entity-bridge reasoning in scientific Wikipedia retrieval.

Query and Relevance Type Tendencies

Queries ask about Swiss physicists, food preservation, edible fungi, basidiomycete names, plumbing traps, journals, measurements, and other chemistry-adjacent facts. Positive documents are Wikipedia passages containing the final answer.

The relevance relation is answer support under multi-hop wording. A relevant passage often contains the final fact after the query has pointed through another entity or property.

Representative Failure Modes

Likely failures include retrieving a page that shares the bridge entity but not the answer, confusing related scientific organisms or people, and selecting a topically adjacent chemistry page instead of the final supporting passage.

BM25 is vulnerable to bridge-term distraction. Dense retrieval can still over-match broad scientific relatedness. Hybrid retrieval helps by combining rare term anchors with semantic intent.

Training Data That May Help

Useful training data includes non-overlapping ChemHotpotQA train retrieval pairs, HotpotQA multi-hop retrieval with supporting facts, chemistry-filtered Wikipedia QA pairs, and bridge-entity hard negatives.

Synthetic data should generate HotpotQA-style multi-hop questions from non-evaluation chemistry-related Wikipedia passages. The positive should contain the final answer fact, while negatives may share only the bridge term.

Model Improvement Notes

Strong systems should represent both bridge entities and the requested final property. Hybrid retrieval is the best observed profile here, but dense retrieval is nearly tied and has perfect recall@100. Rerankers should be trained to distinguish bridge pages from final answer pages.

Example Data

Source Reference Table

Representative query and positive source snippets:

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: ChemHotpotQARetrieval test split derived from HotpotQA
• Train/eval overlap audit: not_audited
• Leakage note: exclude ChemHotpotQA test queries, qrels, and positive Wikipedia passages
• Multi-positive training: single_positive_question_document_focus
• Useful training data: non-overlapping ChemHotpotQA train retrieval pairs, HotpotQA multi-hop retrieval with supporting facts, chemistry-filtered Wikipedia question-passage pairs, multi-hop scientific QA with bridge-entity hard negatives