NanoCMTEB / t2

Overview

NanoCMTEB t2 is a Chinese web passage retrieval task based on the T2Ranking benchmark family. Queries are short Chinese search strings, and documents are noisy web passages that may include copied text, HTML fragments, article sections, product text, medical notes, or other web content. The task measures ranked retrieval quality when most queries have several relevant passages.

Details

What the Original Data Measures

T2Ranking is a large-scale Chinese passage ranking benchmark built from real search-engine queries, expert relevance labels, diversified candidates, and multiple relevance levels. C-MTEB includes T2Retrieval as part of its Chinese retrieval group.

The task is closer to web passage ranking than single-answer QA. A query may be short and ambiguous, and several passages can be relevant at different degrees. The retriever must rank passages that satisfy the search intent above noisy or merely term-overlapping candidates.

Observed Data Profile

The task contains 200 queries, 10,000 documents, and 979 relevance judgments. It is strongly multi-positive: there are 4.90 positives per query on average, a minimum of 1, a median of 4.0, a maximum of 23, and 175 multi-positive queries, or 87.50% of the set.

Queries average 10.74 Chinese characters. Documents average 913.50 characters, making them long for this NanoCMTEB group. Documents may contain boilerplate, copied page fragments, HTML artifacts, and mixed-quality web text.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.7944, hit@10 of 0.9600, and recall@100 of 0.8662 using the top-500 BM25 candidate subset. This is a strong lexical profile: short web queries often contain highly salient terms that appear in relevant passages.

BM25 is still weaker than dense retrieval because relevant passages may satisfy the intent without exact phrase overlap, while noisy web passages can repeat query terms without being useful. The high hit@10 hides ordering limitations among multiple positives.

Dense Evaluation Profile

The dense harrier-oss-270m run reaches nDCG@10 of 0.9245, hit@10 of 0.9800, and recall@100 of 0.9683. Dense retrieval is the strongest top-ranking profile. It substantially improves over BM25 in nDCG@10 and recall@100.

This shows that embedding similarity is highly effective for T2-style Chinese passage ranking. Dense retrieval can identify answer-bearing or intent-satisfying passages despite noisy formatting and query-passage wording differences.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate set reaches nDCG@10 of 0.8604, hit@10 of 0.9800, and recall@100 of 0.9806. It uses a top-100 candidate range with an optional rank-101 safeguard; this task has 1 safeguard row, candidate counts from 100 to 101, and a mean of 100.01 candidates.

Hybrid retrieval has the best recall@100 and ties dense hit@10, but dense remains clearly better for nDCG@10. The hybrid pool is useful for downstream reranking, while dense retrieval gives the best first-page ordering.

Metric Interpretation for Model Researchers

This task is dense-favorable with a strong sparse baseline. Because most queries have multiple positives, nDCG@10 is more informative than hit@10. A model should rank several highly relevant passages early, not just find one positive.

BM25 demonstrates that exact Chinese query terms are valuable, but dense retrieval better separates useful passages from noisy term-overlap passages. Reranking_hybrid provides broad coverage and is a good candidate pool for rerankers.

Query and Relevance Type Tendencies

Queries include entertainment knowledge, geography, handwriting, traditional medicine, legal or consumer questions, and practical web-search topics. Positive documents are web passages with answer-bearing content, sometimes mixed with formatting artifacts or copied snippets.

The relevance relation is intent satisfaction. A positive passage should answer or strongly satisfy the query, and multiple passages can be valid for the same query.

Representative Failure Modes

Likely failures include retrieving pages that repeat entity names without answering, under-ranking cleaner answer passages behind noisy copied text, missing paraphrased answers, and failing to order multiple positives by usefulness.

BM25 is vulnerable to noisy exact-match passages. Dense retrieval can still over-match broad topic. Hybrid retrieval improves coverage but may need reranking to recover dense's top-order quality.

Training Data That May Help

Useful training data includes non-overlapping T2Ranking relevance data, Chinese web search passage-ranking annotations, multi-positive retrieval training data, and relevance-graded hard negatives.

Synthetic data should start from noisy Chinese web passages and create short search queries with multiple relevant passages. Hard negatives should preserve query terms while differing in answer specificity, relevance grade, or intent.

Model Improvement Notes

Strong systems should handle short Chinese queries, noisy web passages, and multi-positive ranking. Dense retrieval is the strongest observed first-stage method, while hybrid retrieval can provide higher recall for reranking.

The task is useful for evaluating Chinese web ranking models where top-10 ordering among several relevant passages matters.

Example Data

Source Reference Table

Representative query and positive source snippets:

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: available
• Evaluation split origin: T2Retrieval dev
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoCMTEB t2 queries, qrels, and web passages
• Multi-positive training: preserve_multiple_relevant_passages_per_query
• Useful training data: non-overlapping T2Ranking relevance data, Chinese web search passage-ranking annotations, multi-positive retrieval training data, relevance-graded hard negatives