NanoSSRB / ProductSearch

Overview#

ProductSearch evaluates attribute-aware retrieval over JSON records for goods and services, including vehicles, books, restaurants, electronics, and other catalog entities. It contains 200 English queries, 10,000 documents, and 600 positive qrels. Requests combine category, brand, price, date, availability, rating, opening hours, capacity, or technical attributes with softer preferences such as reliability, popularity, safety, or suitability. A relevant result must meet the full shopping intent, not merely describe a related product.

Details#

What the Original Data Measures#

SSRB studies direct natural-language querying of semi-structured objects. ProductSearch is the clearest consumer-search instance: structured filters and subjective needs appear together in one request.

Observed Data Profile#

Queries average 232 characters and documents 628 characters, with three positives per query. Examples ask for restaurants open on a day, novels after a date, or vehicles satisfying price, capacity, safety, fuel, and category rules.

BM25 Evaluation Profile#

BM25 reaches 0.3839 nDCG@10, 0.7250 hit@10, and 0.8783 Recall@100. Brands, categories, cuisines, and attributes provide strong lexical anchors, but subjective preferences and numeric filters remain error-prone.

Dense Evaluation Profile#

Dense retrieval is strongest at 0.5305 nDCG@10, 0.8050 hit@10, and 0.9583 Recall@100. It captures suitability and product-intent paraphrases well, though it may still rank an attractive item that violates one hard filter.

Reranking Hybrid Evaluation Profile#

Hybrid reaches 0.5132 nDCG@10, 0.8200 hit@10, and 0.9633 Recall@100. It gives the best coverage and hit rate, while dense retains slightly better top-ten ordering on this candidate profile.

Metric Interpretation for Model Researchers#

Recall@100 measures catalog coverage before reranking; nDCG@10 measures whether the most compliant items surface first. Dense strength indicates semantic shopping intent matters. Compare dense and hybrid to determine whether lexical attributes add useful candidates or introduce near-match noise.

Query and Relevance Type Tendencies#

Queries range from simple category-plus-hours filters to long multi-attribute requests with exclusions and subjective priorities. Relevant items satisfy all hard constraints and plausibly meet the semantic preference expressed by the query.

Representative Failure Modes#

Models may return the right product family at the wrong price, ignore a day or availability field, treat review praise as proof of a missing attribute, or overweight one appealing semantic property. Nested attributes and units create additional relation errors.

Product-Search Notes#

Normalize currencies, dates, units, rating scales, category aliases, and brand names without erasing distinctions. Reviews can support fuzzy preferences but should not override authoritative structured fields.

Training and Leakage Notes#

Exclude evaluation catalog objects, queries, and qrels, and disclose SSRB exposure. Product feeds are duplication-heavy, so audit normalized attributes and descriptions rather than relying only on IDs.

Model Improvement Hints#

Combine semantic product embeddings with explicit attribute matching and constraint-aware reranking. Use hard negatives from the same category that violate exactly one price, date, capacity, availability, or exclusion rule.

Training Data That May Help#

Non-overlapping shopping-query logs, catalog search pairs, product QA, and attribute extraction data can help when licensing and leakage are controlled.

Synthetic Data Guidance#

Generate independent catalogs and grounded shopping requests with realistic attributes and preferences. Create single-filter counterfactuals; never use evaluation products or queries as seeds.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#