NanoSSRB / ResumeSearch

Overview#

ResumeSearch evaluates retrieval over long JSON candidate and resume records. Its 200 English queries search 10,000 documents with 647 positive qrels. Hiring requirements combine roles, years of experience, skills, tools, education, certifications, domain achievements, availability, and explicit exclusions. Evidence is distributed across nested employment, project, education, and skill fields, so the task tests both semantic candidate matching and exact qualification checking. A topically plausible candidate who lacks one mandatory skill or minimum duration is a hard negative.

Details#

What the Original Data Measures#

The SSRB paper's exact-plus-fuzzy retrieval formulation is especially visible in hiring search: broad fit and hard qualifications must be evaluated together across heterogeneous profile schemas.

Observed Data Profile#

ResumeSearch has the longest queries (443 characters on average) and documents (1,356 characters), with 3.24 positives per query. Examples combine campaign outcomes and marketing tools, teaching background and exclusions, or degrees, cloud skills, experience, and start dates.

BM25 Evaluation Profile#

BM25 reaches 0.3297 nDCG@10, 0.6950 hit@10, and 0.8825 Recall@100. Skill names, degrees, titles, and tools anchor candidates well, but lexical overlap does not verify duration, achievement, negation, or equivalent experience.

Dense Evaluation Profile#

Dense retrieval reaches 0.3184 nDCG@10, 0.6850 hit@10, and 0.8377 Recall@100. Semantic role matching helps with paraphrases but loses some exact skill and credential coverage relative to BM25.

Reranking Hybrid Evaluation Profile#

Hybrid is strongest at 0.3896 nDCG@10, 0.7750 hit@10, and 0.9196 Recall@100. The substantial gain shows that resume search benefits from retaining exact skills while adding semantic evidence about duties and achievements.

Metric Interpretation for Model Researchers#

Recall@100 measures whether qualified candidates survive the first stage; nDCG@10 measures whether the best-qualified profiles rank early. Hybrid gains should motivate lexical-semantic fusion. With multiple valid candidates, partial recall can still support screening but does not represent complete candidate discovery.

Query and Relevance Type Tendencies#

Queries include mandatory skills, minimum years, education, certifications, tools, industry experience, quantified outcomes, availability, and negative requirements. Relevant resumes provide evidence for the combined role rather than merely repeating the job title.

Representative Failure Modes#

Typical errors include counting overlapping jobs twice, matching a skill only mentioned in passing, confusing education with work experience, ignoring an exclusion, or overlooking availability. Long profiles can bury disqualifying details far from strong semantic evidence.

Resume-Search Notes#

Preserve skill and certification names, dates, durations, quantities, and negation. Models should connect synonymous job duties while keeping claimed experience distinct from requirements, interests, or unrelated project text.

Training and Leakage Notes#

Exclude NanoSSRB profiles, queries, qrels, and transformed source-test records. Disclose SSRB exposure and use privacy-safe, licensed, or synthetic resumes; benchmark improvement does not justify collecting sensitive personal data.

Model Improvement Hints#

Use long-record field pooling, timeline-aware duration features, requirement-to- evidence alignment, and hard negatives missing one mandatory qualification. Rerankers should produce constraint-level evidence rather than one global fit score.

Training Data That May Help#

Independent job-to-resume matching, skill normalization, career taxonomy, and qualification extraction data can help when privacy and overlap are controlled.

Synthetic Data Guidance#

Generate fictional resumes and grounded job requirements with explicit timelines, skills, achievements, and exclusions. Create counterfactual profiles that fail one requirement; never seed from evaluation resumes.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#