NanoMTEB-French / syntec

Overview

syntec is a French legal and workplace-policy retrieval task built from the Syntec collective bargaining agreement. Queries are natural employee-style questions, and documents are article-level provisions from the agreement. The Nano split contains 100 queries, 90 documents, and 100 positive qrels, with one positive article per query. Documents average 1,226.27 characters, so the model must often identify the relevant clause inside a structured article rather than only match a title. The task is compact, but it is useful for evaluating French retrieval models on semi-legal domain text where terminology, abbreviations, article numbering, and policy paraphrase all matter.

Details

What the Original Data Measures

MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis introduced SyntecRetrieval as a French retrieval dataset. The source material is the Syntec collective bargaining agreement, with articles used as documents and queries written to test article retrieval. The domain is legal and employment-policy oriented, but the wording is often closer to workplace guidance than to highly specialized statutory text.

In retrieval terms, the task measures whether a model can map a practical question about employment conditions to the governing agreement article. The positive article may contain formal article headings, modification notes, exceptions, and lists, while the query may use everyday employee wording.

Observed Data Profile

The Nano split has 100 French queries, 90 documents, and 100 positive judgments. Queries average 72.80 characters, and documents average 1,226.27 characters. All queries have exactly one positive. The small document pool means each candidate list covers the full corpus, but top-rank ordering is still meaningful because several articles can share policy vocabulary.

Questions cover topics such as severance, patents, paid leave, overtime, medical checks after foreign travel, seniority, Sunday work, and classifications. Documents preserve article numbers and formal clause structure, so retrieval quality depends on both lexical legal terms and semantic alignment between a question and the applicable provision.

BM25 Evaluation Profile

BM25 performs reasonably well, reaching nDCG@10 of 0.7180, hit@10 of 0.8900, and recall@100 of 1.0000. Because the corpus has only 90 documents, every positive is available within the candidate list; the main issue is whether BM25 orders the right article near the top. Lexical matching helps when the query contains terms such as "licenciement", "conges", "heures supplementaires", or "deplacement a l'etranger" that also appear in the agreement.

BM25 is weaker when an employee question paraphrases a legal concept, uses an abbreviation, or asks about a practical condition that is expressed formally in the article. The task therefore remains a useful lexical-vs-semantic diagnostic even though the candidate pool is small.

Dense Evaluation Profile

The dense harrier-oss-270m candidates are strongest on this task, with nDCG@10 of 0.8660, hit@10 of 0.9700, and recall@100 of 1.0000. Dense retrieval appears to benefit from mapping practical questions to formal policy language. This is exactly the kind of paraphrase that a pure term-frequency method can miss: an employee may ask what they are entitled to, while the article states conditions, exceptions, and compensation rules in a different style.

For model researchers, syntec is a small but clean signal that semantic retrieval can add value in French domain-specific text. A dense model that does well here likely captures policy concepts, not only surface article titles.

Reranking Hybrid Evaluation Profile

The reranking_hybrid profile reaches nDCG@10 of 0.8463, hit@10 of 0.9800, and recall@100 of 1.0000. It is close to dense retrieval and has the highest hit@10, but dense keeps a slight advantage in nDCG@10. There are no safeguard rows because all 90 documents fit inside the hybrid candidate depth.

This is a case where hybrid search is robust but not clearly dominant. The lexical component preserves exact legal terminology, while the dense component helps bridge paraphrase. The final top ordering still depends on whether the candidate scoring emphasizes the exact governing article rather than a semantically adjacent article.

Metric Interpretation for Model Researchers

syntec is dense-favorable at top-10 ranking, with BM25 still strong and all methods achieving full recall by 100. Because recall is saturated, the key metric is nDCG@10: it measures whether the model ranks the governing article before related but incorrect provisions. The gap between BM25 and dense indicates that semantic matching is important for workplace-policy questions, but the high BM25 baseline shows that terminology overlap remains valuable.

The single-positive setup makes false positives easy to diagnose. If a model ranks an article about a nearby policy area above the positive article, the error is not a recall failure but a legal/semantic discrimination failure.

Query and Relevance Type Tendencies

Queries are practical French questions from an employee or employer perspective. They ask about rights, limits, obligations, and conditions. Positive documents are formal articles containing the clause that resolves the question.

Relevance is article-level. A document may mention the same topic but be wrong if it governs a different condition or worker category. This makes adjacent article negatives especially useful for training and evaluation analysis.

Representative Failure Modes

BM25 can over-rank articles with shared terms but the wrong legal condition. Dense retrieval can over-generalize policy concepts and rank a semantically related article that does not answer the specific question. Both methods can be confused by abbreviations, article modification notes, and long documents where the relevant clause is only a small part of the text.

Hybrid retrieval reduces some of these risks by combining exact legal terms and semantic paraphrase, but it still needs a reranker or stronger scoring model to resolve fine-grained article distinctions.

Training Data That May Help

Useful training data includes French collective-agreement QA pairs, French labor-law FAQ to article retrieval data, non-overlapping employment-policy question-article pairs, and hard negatives from adjacent agreement articles. Training should exclude Nano queries, qrels, and Syntec article positives used in this evaluation.

Synthetic data should preserve article numbers, clause structure, exceptions, and modification notes. Questions should be natural employee wording about leave, seniority, Sunday work, travel, classifications, termination, invention rights, and overtime.

Model Improvement Notes

Strong models should handle both legal terminology and everyday paraphrase. Dense encoders should be trained with domain-specific hard negatives so that they distinguish the governing article from nearby provisions. Rerankers should pay attention to conditions and exceptions, because those details often decide which article is actually relevant.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_found
• Evaluation split origin: test
• Train/eval overlap audit: not_audited
• Leakage note: exclude Nano queries, qrels, and Syntec article positives used in this evaluation
• Multi-positive training: single_positive_question_document_focus
• Useful training data: French collective-agreement QA pairs, French labor-law FAQ to article retrieval data, non-overlapping employment policy question-article pairs, hard negatives from adjacent agreement articles