NanoSSRB / HumanResources

Overview#

HumanResources evaluates retrieval over employee, organization, role, payroll, benefit, feedback, policy, and workplace-activity JSON. The task contains 200 English queries, 10,000 documents, and 578 positive qrels. Queries mix exact employee or department fields, dates, hours, ratings, tenure, and benefit amounts with semantic conditions such as urgency, collaboration, or key-person status. Relevant records must satisfy nested policy and employee constraints, making same-department or same-topic matches insufficient.

Details#

What the Original Data Measures#

SSRB evaluates whether neural retrievers can follow complex natural-language conditions across varying schemas. HR makes those conditions operational and often distributes them across nested employee, project, and benefit fields.

Observed Data Profile#

Queries average 321 characters and documents 359 characters, with 2.89 positives per query. Examples target overtime on critical projects, positive feedback in a date window, or benefits conditioned on tenure and coverage.

BM25 Evaluation Profile#

BM25 reaches 0.1704 nDCG@10, 0.4000 hit@10, and 0.6332 Recall@100. Employee IDs, departments, and policy terms anchor search, but indirect performance or urgency conditions are poorly represented by lexical overlap.

Dense Evaluation Profile#

Dense retrieval reaches 0.2451 nDCG@10, 0.5550 hit@10, and 0.7249 Recall@100. Semantic matching helps with feedback and role intent but may overlook exact hours, ratings, dates, or eligibility requirements.

Reranking Hybrid Evaluation Profile#

Hybrid is strongest at 0.2776 nDCG@10, 0.5450 hit@10, and 0.7578 Recall@100. Its higher recall shows complementary lexical and semantic candidates, while top-ten misses still reveal incomplete constraint checking.

Metric Interpretation for Model Researchers#

Use Recall@100 to assess whether all eligible records reach the reranker and nDCG@10 to assess final policy compliance. Hybrid gains suggest value from combining identifiers and semantic descriptions. With multiple positives, partial coverage may still give useful results but should not be mistaken for complete employee search.

Query and Relevance Type Tendencies#

Queries combine identity, organization, time, performance, compensation, and policy conditions. Relevant records can be time entries, feedback, benefits, employee profiles, or rules, so the model must identify both schema and values.

Representative Failure Modes#

Errors include matching the right employee but wrong period, treating generic praise as collaboration evidence, ignoring a minimum tenure, or confusing covered amount with another monetary field. Nested fields make relation errors especially likely.

HR-Specific Notes#

Identifiers, employment dates, hours, ratings, and organizational roles should be preserved exactly. Semantic encoders must distinguish evidence about an employee from eligibility rules or descriptions attached to a benefit.

Training and Leakage Notes#

Exclude benchmark records and disclose SSRB-derived training. Use privacy-safe synthetic or properly licensed HR data; do not introduce real sensitive employee information merely to improve benchmark performance.

Model Improvement Hints#

Train schema routing and nested-field alignment together. Use negatives that share employee, department, or policy vocabulary but violate one date, rating, tenure, project, or benefit constraint.

Training Data That May Help#

Independent enterprise-search pairs, policy QA, job catalogs, and synthetic HR record retrieval can help when privacy and overlap are controlled.

Synthetic Data Guidance#

Generate fictional organizations and grounded HR requests across time entries, feedback, benefits, and roles. Add counterfactual records with one failed condition; never seed from evaluation objects.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#