NanoMTEB-Thai / multi_long_doc_th

Overview

NanoMTEB-Thai / multi_long_doc_th is the Thai split of MultiLongDocRetrieval. It is a long-document retrieval stress test: generated Thai questions must retrieve the full Thai document that contains the local evidence used to form the question. The task is linked to the MLDR and M3-Embedding line of work, where multilingual retrieval is evaluated across longer and noisier documents than ordinary passage benchmarks. In this Nano split, document length is the defining feature: each query has one positive document, and the average document is roughly 25,993 characters. The task is therefore useful for studying long-context representation, noisy web text, and the limits of embedding models that compress entire documents into a single vector.

Details

What the Original Data Measures

MultiLongDocRetrieval evaluates whether a retriever can identify a long source document from a question generated from a smaller part of that document. The retrieval target is not the specific paragraph; it is the entire document. This makes the benchmark different from passage retrieval, where the relevant evidence is already localized.

The Thai split combines Thai queries with very long Thai or Thai-heavy documents. Some documents contain web boilerplate, navigation text, mixed-language fragments, or low-quality generated text. A model must connect the query to the relevant local span while ranking the whole document.

Observed Data Profile

The Nano split contains 200 queries, 10,000 documents, and 200 positive qrel rows. Each query has exactly one positive document. Queries average 107.79 characters, and documents average 25,993.27 characters.

The examples show a noisy long-document environment: chemistry exam pages, phone-number or numerology pages, mixed English/Thai web pages, gambling-like pages, and accommodation-listing pages. Some queries include prefixes such as คำถาม: or generated-question phrasing. This is not a clean encyclopedic passage task; it is closer to retrieval over messy long web documents.

BM25 Evaluation Profile

The BM25 candidate subset uses top-500 candidates and reaches nDCG@10 of 0.3684, hit@10 of 0.4200, and recall@100 of 0.5600. BM25 is stronger than dense retrieval on this split, which is consistent with the long-document setting. Very long documents contain many lexical hooks, and generated questions often reuse distinctive phrases from the source text.

The same property also causes false positives. Long noisy pages may share repeated terms, boilerplate, or topic vocabulary without being the labeled source document. BM25 succeeds when rare terms or copied phrases identify the document, but it struggles when common web text or repeated templates dominate the scoring signal.

Dense Evaluation Profile

The dense candidate subset from harrier_oss_v1_270m uses top-500 candidates and reaches nDCG@10 of 0.2125, hit@10 of 0.2950, and recall@100 of 0.5350. Dense retrieval is weaker than BM25 in top-rank quality and slightly weaker in recall@100. This suggests that single-vector dense representations have difficulty preserving the local evidence signal inside very long, noisy documents.

For model researchers, this is an important negative case for ordinary embedding retrieval. A question may correspond to a small paragraph or phrase buried in tens of thousands of characters. If the document embedding is dominated by boilerplate, unrelated sections, or broad topic drift, semantic similarity to the query becomes diluted.

Reranking Hybrid Evaluation Profile

The reranking_hybrid subset uses top-100 candidates, with 71 queries carrying a rank-101 safeguard positive. It reaches nDCG@10 of 0.3672, hit@10 of 0.5050, and recall@100 of 0.6450. The hybrid pool has the best hit@10 and recall@100, while BM25 and hybrid are nearly tied on nDCG@10.

This pattern shows that dense retrieval still contributes complementary candidates even though BM25 is the stronger standalone first-stage system. Hybrid retrieval is therefore the most practical candidate-generation strategy for this task. The high number of safeguard rows also shows that many positives are difficult to keep inside a strict top-100 window without explicit coverage protection.

Metric Interpretation for Model Researchers

This split should be read as a long-document candidate-generation benchmark as much as a ranking benchmark. Recall@100 is especially important because a reranker cannot recover a positive document that the first stage fails to include. nDCG@10 remains important, but the main research signal is whether a retrieval method can keep local evidence visible despite extreme document length.

The single-positive setup makes false positives costly. A long document that is topically similar or shares boilerplate is still wrong unless it is the exact source document.

Query and Relevance Type Tendencies

Queries are longer than ordinary Thai retrieval questions and often look generated. They may ask about a specific detail, phrase, recommendation, or procedure found inside a long source page. Relevant documents are full long documents, not extracted passages.

The relevance relation is source-document identity. The correct document is the one from which the question's evidence was derived, even if other documents contain similar language.

Representative Failure Modes

Common failures include matching repeated boilerplate instead of the source page, losing the relevant paragraph inside the document embedding, retrieving another web page with similar template text, and over-weighting common Thai or mixed-language fragments. Dense systems may compress away the relevant detail; sparse systems may be distracted by copied navigation, spam, or repeated keyword blocks.

Training Data That May Help

Useful data includes Thai long-document retrieval, paragraph-to-document supervision, noisy web document retrieval, and hard negatives sampled from documents with similar boilerplate or topic vocabulary. Training should include long inputs with local answer-bearing spans and should avoid overfitting to clean article structure only.

Model Improvement Notes

This task favors retrieval architectures that preserve local evidence in long documents: passage aggregation, late interaction, multi-vector document representations, chunk-aware indexing, or reranking over selected spans. Single-vector dense models should be evaluated carefully, because average document semantics may not reflect the small span that generated the query.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets