NanoRTEB / NanoFinQA

Overview

NanoFinQA is an English financial evidence retrieval task from NanoRTEB. The query is a short financial question, and the relevant document is the report excerpt or table context needed for numerical reasoning. Each query has one positive document among 380 candidates. This split is unusual because BM25 is the strongest or nearly strongest profile: finance questions often contain exact metric names, years, and company-specific terms. reranking_hybrid closely follows BM25, while dense retrieval is weaker on top-rank quality.

Details

What the Original Data Measures

FinQA was introduced as a dataset for numerical reasoning over financial data. The original benchmark uses expert-authored questions over company reports, evidence, and executable reasoning programs.

RTEB uses the evidence retrieval component. A system must retrieve the financial table or report passage that supports the question before any numerical reasoning can happen.

Observed Data Profile

The Nano split contains 200 queries, 380 documents, and 200 positive qrel rows. Every query has exactly one positive. Queries average 101.45 characters, while documents average 3,918.54 characters.

Example questions ask about differences between high and low sales prices, total impairment costs across years, current recourse debt as a percentage of total recourse debt, average operating profit over several years, and future estimated cash payments due in a specific year.

BM25 Evaluation Profile

The BM25 candidate subset uses the full 380-document pool and reaches nDCG@10 of 0.7330, hit@10 of 0.9250, and recall@100 of 1.0000. BM25 is very strong for this task.

The strength comes from query terms such as financial metrics, years, company names, debt categories, and accounting phrases. These tokens often appear directly in the relevant report excerpt or table. The remaining challenge is choosing the right section when many filings share similar finance vocabulary.

Dense Evaluation Profile

The dense candidate subset from harrier_oss_v1_270m uses the full 380-document pool and reaches nDCG@10 of 0.6051, hit@10 of 0.7600, and recall@100 of 0.9700. Dense retrieval is substantially weaker than BM25 on early rank quality.

This suggests that exact lexical anchoring is more useful than broad semantic similarity for this split. Dense embeddings may connect related financial concepts, but they can underweight precise years, line-item names, and table labels.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses top-100 candidates and does not need the rank-101 safeguard. It reaches nDCG@10 of 0.7309, hit@10 of 0.8900, and recall@100 of 1.0000. Hybrid retrieval nearly matches BM25 on nDCG@10 and has saturated recall@100.

The hybrid result shows that sparse lexical matching remains the dominant signal, while dense retrieval can add some robustness. For reranking, both BM25 and hybrid pools provide full recall at rank 100.

Metric Interpretation for Model Researchers

With one positive per query, nDCG@10 measures how early the supporting report excerpt appears, hit@10 measures whether it appears in the first ten candidates, and recall@100 measures reranker availability.

For NanoFinQA, recall@100 is saturated for BM25 and hybrid retrieval, so improvements should focus on ranking the correct evidence first. Preserving exact numeric and temporal terms is critical.

Query and Relevance Type Tendencies

Queries are short finance questions that often name a metric, period, company, or calculation target. Relevant documents are long excerpts from financial reports, frequently containing tables or dense numeric context.

Relevance is evidence sufficiency for the calculation. A passage about the same company or year is wrong if it does not contain the numbers needed for the question.

Representative Failure Modes

Common failures include retrieving a same-company section with the wrong metric, confusing adjacent years, overranking passages with common accounting terms, and missing table rows that contain the decisive number. BM25 can still be misled by repeated finance vocabulary; dense retrieval can blur precise line items.

Training Data That May Help

Useful training data includes financial report evidence retrieval, SEC filing search, table question answering with supporting evidence, and hard negatives sharing company, year, or metric names. Evaluation questions, report excerpts, and qrels should be excluded.

Model Improvement Notes

Models should retain exact numbers, dates, company names, and financial line-item labels in their representations. Hard negatives should use the same company and period but different accounting fields. Sparse and hybrid retrieval are strong foundations for this split because exact terminology matters.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets