NanoMTEB-BR / MedPTRetrieval
Overview#
MedPTRetrieval is a Brazilian Portuguese medical question-answer retrieval task derived from MedPT. The source collection contains authentic patient questions and clinician answers spanning many conditions and several kinds of health intent, such as symptoms, diagnosis, treatment, and medication. Queries can be informal, underspecified, or written with regional and everyday vocabulary, while relevant documents use more clinical explanations. The retrieval task asks a model to surface an answer that addresses the medical intent rather than a passage that merely repeats a disease name. NanoMTEB-BR samples this source into a compact corpus for fast evaluation while preserving native Brazilian Portuguese interactions instead of translated medical text. Because most queries have very few judged positives, the task makes incorrect intent matches visible: answers about a related condition or a different stage of care can be topically similar but still irrelevant.
Details#
What the Task Measures#
The task measures semantic matching between layperson health questions and clinically informed answers. Important distinctions include symptom versus diagnosis, contraindication versus general treatment, patient context, and the requested type of advice.
Metric Interpretation#
nDCG@10 rewards placing the judged answer close to rank one. Hit@10 reports whether one appears anywhere in the first ten. These are retrieval-quality metrics only; they do not establish clinical correctness or authorize use as a medical decision system.
Training and Leakage Notes#
Train with non-overlapping Portuguese medical QA and carefully selected hard negatives for related conditions. Exclude Nano queries, qrels, positive answers, and upstream MedPT evaluation records. Audit paraphrases and copied web answers, and avoid treating synthetic medical content as verified advice.
Public Sources#
Example Data#
| Query | Positive document |
| Fiz cirurgia de hernia incisional há 40 dias e agora, através de USG descobri que tenho mais uma hernia supra- umbilical e uma na inguinal. É normal ter tantas hernias assim, ou pode ser algum outro problema como ruptura abdominal. O que deve ser investigado? [259 chars] | Dificil afirmar a relacao entre elas, mas provavel que no pre-operatorio nao tenha sido investigado minuciosamente sua parede abdominal. Hoje e mandatorio para tratamento de hernias incisionais a tomografia em pre-operatorio, evitando que possiveis defeitos associados nao sejam tratados.Sugiro avaliacao com cirurgiao especialista em parede abdominal. Fique a vontade para entrar em contato caso queira esclarecer e aprofundar melhor no assimto [445 chars] |
| Podemos fazer corrida mesmo tendo esporão? [42 chars] | Na fase aguda, não é recomendado, piora a dor, mas depois de analgésicos e muito alongamento, quando você estiver sem dor, sim, pode, sem problemas. [148 chars] |
| Perdi um casal de filhos com holoprosencefalia. Devo engravidar de novo? Nenhum exame apontou nada. Meu marido e eu temos os mesmos tataravós. [142 chars] | Esta relação familiar sua com seu marido favorece o aparecimento de doenças genéticas do tipo recessivas. Não necessariamente vocês terão um novo filho com a mesma doença, mas acredito que sempre existirá uma possibilidade. Sugiro uma consulta com médico geneticista para que vocês façam um heredograma, testes genéticos, afim de quantificarem melhor esse risco e poderem avaliar se querem correr o risco, ou mesmo, conhecerem alternativas.Espero tê-la(o) ajudado. [464 chars] |
Dataset Information#
| Field | Value |
| Nano set | NanoMTEB-BR |
| Backing dataset | NanoMTEB-BR |
| Task / split | MedPTRetrieval |
| Hugging Face dataset | hakari-bench/NanoMTEB-BR |
| Language | pt |
| Category | natural_language |
| Queries | 200 |
| Documents | 500 |
| Positive qrels | 204 |
| Positives / query avg | 1.02 |
| Positives / query min | 1 |
| Positives / query median | 1.00 |
| Positives / query max | 3 |
| Multi-positive queries | 3 (1.50%) |
| Query length avg chars | 151.97 |
| Document length avg chars | 368.21 |
Candidate Subsets#
| Profile | Config | nDCG@10 | Hit@10 | Recall@100 | Candidates |
| BM25 | bm25 | 0.4328 | 0.5550 | 0.7794 | top-500 |
| Dense | harrier_oss_v1_270m | 0.6974 | 0.8200 | 0.9755 | top-500 |
| Reranking hybrid | reranking_hybrid | 0.5975 | 0.7300 | 0.9657 | top-100 |
Hugging Face Links#
- Nano dataset: hakari-bench/NanoMTEB-BR
- MedPT paper: https://arxiv.org/abs/2511.11878
- MedPT dataset: https://huggingface.co/datasets/AKCIT/MedPT
Source Reference Table#
| Title | Year | Type | URL |
| MedPT: A Massive Medical Question Answering Dataset for Brazilian-Portuguese Speakers | 2025 | paper | https://arxiv.org/abs/2511.11878 |
| AKCIT/MedPT | dataset page | https://huggingface.co/datasets/AKCIT/MedPT |