NanoDAPFAM / NanoDAPFAMInTitlAbsToFullText

Overview

NanoDAPFAMInTitlAbsToFullText is an English patent-family retrieval task. The query contains a source patent family's title and abstract, and the target document contains the full text of a candidate patent family. Relevance is restricted to DAPFAM IN-domain positives, so target families are citation-related and share IPC3 technical class with the query family.

This split tests compact summary-to-full-text retrieval inside a shared patent domain. The query gives only summary-level invention information, while targets are very long patent records. The model must expand from a title and abstract to same-domain prior-art families whose relevant evidence may appear anywhere in full text.

Details

What the Original Data Measures

DAPFAM is a domain-aware patent-family retrieval benchmark that uses citation links as relevance labels. IN-domain positives are citation-related families sharing IPC3 domain with the source. This split uses short source fields and long target fields to test in-domain prior-art retrieval from concise search input.

Observed Data Profile

This Nano split contains 200 queries, 10,000 documents, and 3,072 positive qrels. There are 194 multi-positive queries, with an average of 15.36 positives per query, a minimum of 1, median of 18.0, and maximum of 20. Queries average 771.27 characters, while full-text targets average 68,924.25 characters.

The query side resembles a patent search summary. The target side contains claims, descriptions, and background material, so ranking must tolerate substantial full-text noise.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.3490, hit@10 of 0.8200, and recall@100 of 0.5677 with a top-500 candidate pool. Title and abstract terms remain useful because in-domain patents often reuse technical vocabulary. BM25 can often find at least one positive.

The limitation is full-text length and same-domain ambiguity. Many target documents share the same vocabulary, and long descriptions can contain query terms incidentally. BM25 therefore trails dense and hybrid methods in recall.

Dense Evaluation Profile

The dense harrier-oss-270m profile reaches nDCG@10 of 0.4255, hit@10 of 0.8750, and recall@100 of 0.6729. Dense retrieval improves over BM25 by matching the summary-level invention concept to longer target-family content.

Dense retrieval is useful when target patents describe the same technical contribution with different phrasing. It is still limited by long-document representation and multi-positive coverage.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset reaches nDCG@10 of 0.4228, hit@10 of 0.8900, and recall@100 of 0.6803. It uses top-100 candidates with optional rank-101 safeguards; five rows contain 101 candidates and five safeguard-positive rows are recorded. Hybrid retrieval gives the best hit@10 and recall@100, while dense retrieval has slightly higher nDCG@10.

This means exact summary terms and semantic similarity are complementary. Hybrid is the better candidate pool for downstream reranking, while dense ranking remains very competitive at the top.

Metric Interpretation for Model Researchers

This is an in-domain summary-to-full-text task. High hit@10 does not imply complete retrieval because each query has many positives. Recall@100 shows that dense and hybrid methods recover substantially more cited families than BM25.

The split is useful for evaluating long patent target handling from short user-facing patent summaries.

Query and Relevance Type Tendencies

Queries are titles and abstracts. Documents are full patent texts. Positives are same-domain cited families. Since the domain is shared, broad topic classification is not enough; models must rank citation-relevant families within a technical class.

Representative Failure Modes

BM25 may retrieve full texts with incidental term overlap. Dense retrieval may rank semantically related but non-cited same-domain families. Hybrid retrieval can still be distracted by long target documents whose background sections match the query.

Training Data That May Help

Useful training data includes same-domain title-abstract patent retrieval, full-text patent prior-art search, and family-level citation retrieval. Training should exclude NanoDAPFAM evaluation family IDs and citation labels.

Synthetic data should pair title-abstract summaries with full-text same-domain patent family records, using cited families as positives and same-IPC non-cited families as hard negatives.

Model Improvement Notes

Improving this task requires summary-to-long-document matching inside the same patent domain. Models should identify the main inventive concept from title and abstract, then locate related evidence in very long target records.

For reranking, passage-level evidence aggregation and citation-likelihood signals may improve positive coverage.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_confirmed
• Evaluation split origin: DAPFAM IN-domain title-abstract to full-text patent-family retrieval
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoDAPFAM evaluation family IDs and citation labels
• Multi-positive training: citation_family_multi_positive
• Useful training data: same-domain title-abstract patent retrieval, full-text patent prior-art search, family-level citation retrieval