NanoR2MED / NanoR2MEDPMCClinical

Overview

NanoR2MEDPMCClinical is an English clinical case retrieval task from R2MED. Queries are concise patient case descriptions, and documents are longer PubMed Central case presentations. Relevance is based on similar cases, usually tied by diagnosis or clinically meaningful case similarity. Unlike several other NanoR2MED tasks, BM25 is stronger than dense retrieval here, likely because case reports repeat distinctive anatomy, disease, imaging, and symptom terms. The reranking_hybrid pool is strongest overall, combining lexical case-report anchors with semantic similarity.

Details

What the Original Data Measures

R2MED defines clinical case retrieval as retrieving cases with the same diagnosis or clinically useful similarity. The benchmark paper describes collecting patient records from PMC-Patients and IIYi sources, extracting clinical presentations, and using confirmed diagnosis as the reasoning bridge between query and relevant cases.

For PMC-Clinical, the source data comes from PubMed Central case reports. R2MED identifies multi-case articles, uses one case as a query source, and treats related cases as positives after filtering and review. The task is intended for research evaluation, not clinical decision-making.

Observed Data Profile

The Nano split contains 114 queries, 10,000 documents, and 248 positive qrel rows. Queries average 827.68 characters, and documents average 2,103.50 characters. Documents are substantially longer than queries because they are case presentations.

Each query has 2.18 positives on average, with a median of 2 and a maximum of 4. Multi-positive queries account for 83 of 114 queries, or 72.81%. Examples include mandibular osteolytic lesions, melanoma-associated intracranial events, retroperitoneal tumors, skull-base inflammatory disease, and splenic masses.

BM25 Evaluation Profile

The BM25 candidate subset uses top-500 candidates and reaches nDCG@10 of 0.3933, hit@10 of 0.7105, and recall@100 of 0.8185. BM25 is a strong baseline for this split and outperforms dense retrieval on all reported metrics.

This reflects the nature of PMC case reports. Diagnoses, anatomy, imaging terms, pathology descriptors, and rare disease names often appear explicitly in both query and relevant cases. Exact term frequency is therefore highly informative.

Dense Evaluation Profile

The dense candidate subset from harrier_oss_v1_270m uses top-500 candidates and reaches nDCG@10 of 0.3584, hit@10 of 0.6316, and recall@100 of 0.7540. Dense retrieval is useful but weaker than BM25. It can capture case similarity across paraphrase, but it may overgeneralize among cases that share clinical context without sharing the correct diagnosis.

This split is a reminder that dense models do not automatically dominate in biomedical retrieval. When rare clinical entities and anatomy terms are repeated, lexical matching can be the more reliable first-stage signal.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses top-100 candidates, with six rows receiving the optional rank-101 safeguard. It reaches nDCG@10 of 0.4477, hit@10 of 0.7807, and recall@100 of 0.8589. This is the strongest profile for the split.

Hybrid retrieval benefits from both sides: BM25 supplies exact disease and anatomy anchors, while dense retrieval adds cases that are clinically related despite phrasing differences. The combined pool is therefore the best target for reranking.

Metric Interpretation for Model Researchers

This is a multi-positive similar-case task. nDCG@10 rewards ranking multiple related cases early, hit@10 measures whether at least one related case is retrieved in the first ten results, and recall@100 measures candidate coverage for reranking.

For PMC-Clinical, BM25 is the stronger standalone baseline, dense retrieval is the semantic comparison point, and hybrid retrieval is the best overall candidate-generation profile. A successful model should preserve rare clinical terms while improving diagnosis-level case matching.

Query and Relevance Type Tendencies

Queries are case summaries with age, symptoms, imaging, pathology, clinical course, or diagnostic findings. Relevant documents are longer PMC case presentations. Positives may share diagnosis or clinically meaningful similarity without matching every demographic or symptom detail.

Relevance is case-similarity oriented. A passage can share a symptom or anatomical region and still be wrong if it reflects a different diagnosis.

Representative Failure Modes

Failures include retrieving a case from the same anatomical region but wrong diagnosis, overmatching broad neurocognitive or dental symptoms, ranking general case-report passages above rare-disease matches, and missing cases where the relevant diagnosis is stated with different terminology. Dense retrieval may cluster by broad specialty; BM25 may overvalue repeated anatomy terms.

Training Data That May Help

Useful training data includes non-overlapping clinical case similarity pairs, PMC-Patients retrieval, diagnosis-labeled case matching, clinical entity linking from case descriptions, and hard negatives sharing symptoms but not diagnoses. Evaluation cases, qrels, positive case passages, and same-article duplicates should be excluded.

Model Improvement Notes

Models should jointly handle rare disease terms, anatomy, imaging, pathology, and diagnosis-level similarity. Multi-positive objectives are appropriate because many queries have more than one related case. Hard negatives should share symptoms or anatomy while differing in diagnosis.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_found
• Evaluation split origin: R2MED benchmark release sampled into NanoR2MED
• Train/eval overlap audit: not_audited
• Leakage note: exclude R2MED PMC-Clinical evaluation cases, qrels, positive case passages, and same-article duplicates
• Multi-positive training: multi_positive_objective
• Useful training data: non-overlapping clinical case similarity pairs, PMC-Patients retrieval and diagnosis-labeled case matching, clinical entity linking from case descriptions, hard negatives sharing symptoms but not diagnoses