NanoSSRB / FinanceAndEconomics

Overview#

FinanceAndEconomics evaluates retrieval over JSON records for banking, transactions, markets, insurance, budgets, risk, personal finance, and economic indicators. Its 200 English queries search 10,000 documents with 601 positive qrels. Requests frequently combine currencies, amounts, dates, account or asset types, geographic conditions, and qualitative risk or purpose descriptions. The main difficulty is precise conjunction: a transaction can share the right currency and topic yet fail an amount threshold or beneficiary condition.

Details#

What the Original Data Measures#

The SSRB paper tests neural retrieval as a direct interface to semi-structured collections. This domain emphasizes numerical and logical semantics that keyword search cannot reliably enforce.

Observed Data Profile#

Queries average 314 characters and documents 556 characters, with about three positives per query. Examples combine budgets, portfolio composition, crypto amounts, suspicious-transfer intent, or economic conditions with hard filters.

BM25 Evaluation Profile#

BM25 reaches 0.1532 nDCG@10, 0.3950 hit@10, and 0.5524 Recall@100. Currency, ticker, account, and transaction vocabulary helps, but token overlap does not perform numerical comparison or reliably infer financial intent.

Dense Evaluation Profile#

Dense retrieval reaches 0.2795 nDCG@10, 0.6200 hit@10, and 0.6972 Recall@100. It better captures risk and purpose paraphrases, while exact amounts, bounds, and currencies remain frequent failure points.

Reranking Hybrid Evaluation Profile#

Hybrid reaches 0.2791 nDCG@10, 0.6350 hit@10, and 0.6889 Recall@100. It slightly improves hit rate but does not recover every dense candidate, showing that both candidate coverage and constraint-aware ordering remain open problems.

Metric Interpretation for Model Researchers#

This is the lowest-recall NanoSSRB domain, so Recall@100 is as important as nDCG@10. A reranker cannot repair a missing qualifying transaction. Dense gains show value from semantic intent, but high scores require reliable comparison of numbers, dates, currencies, and nested financial attributes.

Query and Relevance Type Tendencies#

Queries include threshold search, temporal windows, portfolio suitability, beneficiary or geography filters, and risk interpretation. Relevant records must satisfy hard financial constraints and the softer requested rationale.

Representative Failure Modes#

Common errors include reversing greater-than and less-than, confusing account and transaction currencies, ignoring a date window, or matching generic risk language without the requested asset allocation. Near-identical records with one wrong numeric field are strong hard negatives.

Finance-Specific Notes#

Amounts, percentages, currency codes, timestamps, and negative conditions must remain distinct. Models should connect aliases such as ETH/Ethereum while not conflating units or treating all large-number mentions as equivalent.

Training and Leakage Notes#

Exclude evaluation objects and SSRB test derivatives; disclose SSRB exposure. Synthetic financial data should not copy real personal identifiers or seed from the benchmark's positive records.

Model Improvement Hints#

Combine field-aware embeddings with explicit numeric/date features and constraint-level reranking. Train on counterfactual negatives that change one amount, currency, period, beneficiary, or risk attribute.

Training Data That May Help#

Non-overlapping transaction search, filing evidence, market-data catalogs, and financial QA retrieval pairs can teach terminology and constraint grounding.

Synthetic Data Guidance#

Create independent ledger, budget, portfolio, insurance, and indicator schemas; generate answerable requests and single-constraint near misses without using evaluation queries or positives.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#