NanoRTEB

Overview

NanoRTEB is the Nano task group for the open English portion of RTEB, the Retrieval Embedding Benchmark. It covers 14 retrieval tasks across legal, finance, code, healthcare, and technical-document settings. The group is intentionally heterogeneous: some tasks retrieve statutes or case law from long legal fact patterns, some retrieve financial filing evidence, some retrieve code or SQL from programming questions, and some retrieve medical or developer support answers.

The group contains 2,390 queries, 33,864 task-local documents, and 9,150 positive qrel rows. It evaluates retrieval-first behavior across practical domains rather than one academic task family. Models must handle long queries, short queries, long documents, code snippets, SQL strings, tables, multi-positive medical evidence, and enterprise-style technical search.

What This Group Measures

RTEB focuses on realistic retrieval evaluation across law, healthcare, code, and finance. NanoRTEB preserves that practical-domain mixture. Legal tasks retrieve case law, statutes, and legal clauses. Finance tasks retrieve filing evidence, financial answer passages, or table-backed numerical evidence. Code tasks retrieve Python solutions, data-science code, or SQL queries. Healthcare tasks retrieve patient responses or clinical evidence. FreshStack retrieves technical documentation for developer questions.

Several tasks are repurposed from generation or QA datasets, so relevance is not always ordinary passage similarity. A programming problem may retrieve an implementation; a table question may retrieve SQL; a legal summary may retrieve a clause; a clinical question may retrieve multiple biomedical passages. This is the main value of the group for retrieval-model research.

Task Families

• Legal retrieval: NanoAILACasedocs, NanoAILAStatutes, and NanoLegalSummarization.
• Finance retrieval: NanoFinanceBench, NanoHC3Finance, and NanoFinQA.
• Code and structured-query retrieval: NanoApps, NanoDS1000, NanoHumanEval, NanoMBPP, and NanoWikiSQL.
• Healthcare retrieval: NanoChatDoctor and NanoCUREv1.
• Technical-document retrieval: NanoFreshStack.

Dataset Shape

All tasks are English. Query and document length vary substantially. AILA legal queries are long fact patterns, while HC3Finance, MBPP, FinQA, and legal summarization use much shorter queries. Code and WikiSQL tasks often have long problem statements but short target code or SQL. AILA case documents are very long; WikiSQL target documents are short SQL strings.

Positive density also varies. NanoCUREv1 dominates the qrel count with 5,163 positives and 28.37 positives per query. NanoFreshStack, AILA, and legal summarization are also multi-positive. Most code and finance splits are single-positive, so exact top-rank retrieval matters more there.

Retrieval Behavior

BM25 Profile

BM25 is best only for NanoFinQA. That task often shares company names, years, financial metrics, and table labels between the query and evidence, giving sparse retrieval strong anchors. BM25 is also reasonable for NanoCUREv1, NanoLegalSummarization, and NanoWikiSQL, where terminology or schema terms often overlap with the target.

BM25 fails badly on code-generation-style retrieval. NanoApps has 0.0084 nDCG@10, and NanoMBPP has 0.0875, because problem statements and correct implementations share little surface text. Long legal and technical-document tasks also show that finding one overlapping term is not enough; ranking the right precedent, provision, or document requires more than token frequency.

Dense Profile

Dense retrieval with harrier-oss-270m is the strongest group-level profile at 0.5764 nDCG@10. It is best for most single-answer semantic or code tasks, including NanoApps, NanoChatDoctor, NanoDS1000, NanoFinanceBench, NanoHC3Finance, NanoMBPP, and NanoWikiSQL. The gains on code and SQL are large, especially for MBPP and WikiSQL, where dense similarity can connect a natural-language task to an implementation or query.

Dense is not uniformly best. It trails BM25 on NanoFinQA, where exact financial evidence terms are highly useful, and trails hybrid on several multi-positive or evidence-heavy tasks. Still, dense retrieval is the best single profile for the group because many NanoRTEB tasks require semantic matching beyond exact lexical overlap.

Reranking Hybrid Profile

The reranking hybrid profile is best for NanoCUREv1, NanoFreshStack, NanoHumanEval, and NanoLegalSummarization, and it has the best group-level recall@100. These tasks benefit from combining exact anchors with semantic matching: biomedical terms plus clinical meaning, documentation terms plus developer intent, function identifiers plus code behavior, and legal vocabulary plus clause meaning.

Hybrid is less effective than dense on APPS, MBPP, WikiSQL, and FinanceBench, where sparse evidence can add noise to a strong semantic or structured-code signal. The group therefore supports task-aware retrieval design: dense is a strong default, hybrid is useful for evidence-rich and multi-positive tasks, and BM25 remains important for highly lexical financial evidence.

Task Summary

Interpretation Notes for Model Researchers

NanoRTEB is best interpreted by domain and target type. Code and SQL tasks strongly favor dense retrieval. Multi-positive clinical, legal, and technical documentation tasks often benefit from hybrid candidate generation. Finance evidence can remain highly lexical. A single aggregate score can hide whether a model is improving code retrieval, financial filing retrieval, legal search, or healthcare evidence retrieval.

The group is also sensitive to memorization. Several code datasets have exact solutions; legal and finance tasks have small document pools; CURE and FreshStack include many positives. Per-task inspection and leakage audits matter when using NanoRTEB for model comparison.

Training and Leakage Notes

Useful training data includes legal precedent and statute retrieval, contract clause retrieval, SEC filing evidence retrieval, table QA retrieval, problem-to-code and docstring-to-code pairs, text-to-SQL examples, clinical evidence retrieval, patient-question-to-answer ranking, and developer documentation retrieval. Multi-positive tasks should retain multiple support documents when possible.

Leakage control should exclude NanoRTEB evaluation queries, qrels, positive documents, exact code solutions, SQL targets, legal clauses, financial tables, and near-duplicate source records. For code tasks, exact solution memorization is a serious risk; for legal, finance, and healthcare tasks, passage overlap can inflate scores without improving retrieval generalization.

Source Reference Table

Metadata Summary

Task Metadata Summary