NanoSSRB / Academic

Overview#

Academic evaluates natural-language retrieval over JSON records for grants, researchers, publications, collaborations, competitions, workshops, and related academic activities. Its 200 English queries search 10,000 records with 600 positive qrels. A request can combine an exact funder, amount threshold, institution age, deadline, degree, or research field with a semantic preference such as innovation, healthcare relevance, or a strong publication record. Relevant objects must satisfy the combined conditions, so an academically similar record can still be wrong. BM25 benefits from named institutions and funding agencies; dense retrieval handles paraphrased expertise and purpose; hybrid candidates combine both signals.

Details#

What the Original Data Measures#

The SSRB paper defines retrieval over heterogeneous semi-structured objects using queries that mix exact and fuzzy conditions. Academic applies that formulation to multiple academic schemas rather than treating all records as one fixed table.

Observed Data Profile#

Queries average 298 characters and documents average 900 characters. The three positives per query often share a schema but vary in field layout. Examples ask for NIH grants above a threshold, professors with specified expertise and education, or competitions constrained by audience and deadline.

BM25 Evaluation Profile#

BM25 reaches 0.2975 nDCG@10, 0.6100 hit@10, and 0.8433 Recall@100. Exact agency, institution, discipline, and award terms provide good anchors, but lexical matching struggles when suitability is expressed indirectly.

Dense Evaluation Profile#

Dense retrieval reaches 0.4910 nDCG@10, 0.7850 hit@10, and 0.9467 Recall@100. It connects paraphrased research interests and goals, though a semantic match may still violate a numeric, date, or exclusion constraint.

Reranking Hybrid Evaluation Profile#

Hybrid reaches 0.4873 nDCG@10, 0.7900 hit@10, and 0.9433 Recall@100. It improves hit rate slightly over dense while retaining exact anchors, but its top-rank quality remains close to the dense profile.

Metric Interpretation for Model Researchers#

nDCG@10 measures whether fully qualifying records appear early; Recall@100 shows whether the first stage covers all valid alternatives. Dense gains over BM25 indicate that academic intent and expertise paraphrases matter. Inspect constraint failures when recall is high but nDCG is low.

Query and Relevance Type Tendencies#

Queries cover entity lookup, range filtering, temporal conditions, research topic matching, and multi-field qualification. Relevant records satisfy both literal fields and the stated academic purpose; topical neighbors that miss one condition are not interchangeable positives.

Representative Failure Modes#

Typical errors include retrieving the right field but wrong amount, confusing related research areas, ignoring a negated interest, or ranking an expired competition. Long nested records can dilute the one field that disqualifies an otherwise convincing match.

Academic-Domain Notes#

Models should preserve acronyms, institution names, grant identifiers, dates, and money values while linking broader and narrower research terminology. Hierarchy-aware encoding helps distinguish a research interest from a degree, publication title, or project objective.

Training and Leakage Notes#

Exclude NanoSSRB queries, qrels, positives, and source-test objects. Disclose training on SSRB or vec-ai/struct-ir. Near-duplicate synthetic profiles with changed constraints are particularly important to audit.

Model Improvement Hints#

Use field-aware serialization, numeric/date representations, conjunction-aware training, and hard negatives that violate exactly one condition. Rerankers should compare every requested constraint against its corresponding field.

Training Data That May Help#

Independent grant databases, publication search pairs, scholarly profiles, and academic-event catalogs can help when evaluation records are excluded.

Synthetic Data Guidance#

Generate independent academic schemas and grounded requests with explicit and semantic constraints. Include near-matches with the wrong funder, date, amount, topic, or institution; never seed generation from evaluation positives.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#