NanoDAPFAM / NanoDAPFAMAllTitlAbsClmToTitlAbsClm

Overview

NanoDAPFAMAllTitlAbsClmToTitlAbsClm is an English patent-family retrieval task. Both query and target documents include title, abstract, and claims. Relevance comes from DAPFAM citation links under the ALL condition, so positives include both same-domain and cross-domain family relations.

This split is a claim-rich patent retrieval setting. Compared with title-abstract targets, candidate documents expose more technical and legal detail. Compared with full-text targets, they are much shorter and less noisy. The model must use claim language to identify related patent families without being misled by generic claim phrasing.

Details

What the Original Data Measures

DAPFAM benchmarks family-level patent retrieval using citation links and domain-aware IPC3 relations. It aggregates patents into families and represents them with fields such as titles, abstracts, claims, descriptions, and citations. The ALL condition keeps all citation positives, regardless of whether source and target share an IPC3 class.

This split evaluates title-abstract-claims to title-abstract-claims retrieval. It is a balanced field setting: both sides contain summaries and claims, so both lexical claim overlap and semantic invention similarity can contribute.

Observed Data Profile

This Nano split contains 200 queries, 10,000 documents, and 3,989 positive qrels. Every query has multiple positives, with an average of 19.95 positives per query, a minimum of 9, a median of 20.0, and a maximum of 20. Queries average 8,339.47 characters, and target documents average 7,229.06 characters.

The observed examples include patents about snow removal, transportation systems, hollow microspheres, carpets, and vehicle steering. Both query and target sides include claim-style enumerations and technical component descriptions.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.3360, hit@10 of 0.8550, and recall@100 of 0.4415 with a top-500 candidate pool. The score is higher than the title-abstract target variant because target claims provide additional lexical anchors. Shared components, materials, functional terms, and claim phrases can all help lexical retrieval.

BM25 still misses many positives because patent claims are repetitive and formal. Similar legal phrasing can appear across unrelated families, while citation-relevant families may describe related concepts using different claim structures.

Dense Evaluation Profile

The dense harrier-oss-270m profile is strongest, with nDCG@10 of 0.4156, hit@10 of 0.8950, and recall@100 of 0.5530. Dense retrieval improves over BM25 by capturing technical relatedness across paraphrased claim language.

Dense retrieval benefits from having claims on both sides: the target contains enough technical detail to support semantic matching without the extreme length of full text. Still, multi-positive recall remains challenging because each source patent can cite many related families across different technical aspects.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate subset reaches nDCG@10 of 0.3989, hit@10 of 0.8900, and recall@100 of 0.5463. It uses top-100 candidates with optional rank-101 safeguards; seven rows contain 101 candidates and seven safeguard-positive rows are recorded. Dense retrieval is slightly stronger at the top and in recall, while hybrid is close.

The close dense-hybrid relationship indicates that exact claim terms and semantic similarity both matter. Hybrid retrieval helps recover candidates that one signal alone might miss, but dense retrieval remains the cleanest top-rank signal for this ALL split.

Metric Interpretation for Model Researchers

This is a balanced claim-rich patent retrieval task. It is less sparse than title-abstract targets and less noisy than full-text targets. The metric pattern shows that dense retrieval is strongest, but BM25 is meaningful because both sides contain claims.

Because every query has around 20 positives, researchers should inspect both nDCG@10 and recall@100. A model may find one relevant family while missing many other cited families.

Query and Relevance Type Tendencies

Queries and documents include title, abstract, and claims. The ALL condition includes both IN-domain and OUT-domain relations. Positives are citation-style related patent families, not necessarily duplicates or same-class patents.

Claim text tends to contain detailed components, steps, materials, and functional relationships. These features are useful but can also be repetitive across similar patent classes.

Representative Failure Modes

BM25 may over-rank candidates with shared claim boilerplate. Dense retrieval may over-rank broad technical similarity without the citation-style dependency required by qrels. Hybrid retrieval can still struggle when lexical and semantic signals point to different aspects of a multi-positive query.

Another failure mode is domain mixing. In ALL retrieval, cross-domain positives may use different terminology, while same-domain negatives may look lexically closer.

Training Data That May Help

Useful training data includes patent claim retrieval, family-level citation prediction, and prior-art search pairs with claim text. Training should exclude NanoDAPFAM evaluation query and target families and citation labels.

Synthetic data should generate patent titles, abstracts, and claims with related cited family records. Positives should be cited or technically dependent patent families, and hard negatives should share claim terminology without the same inventive relation.

Model Improvement Notes

Improving this task requires claim-aware semantic retrieval. Models should preserve components, process steps, materials, and technical effects while reducing the influence of generic legal phrasing.

For reranking, useful signals include claim-element correspondence, problem-solution alignment, citation-likelihood, and domain-aware interpretation of IPC overlap.

Example Data

Source Reference Table

Dataset Information

Candidate Subsets

Training and Leakage Metadata

• Original train split: not_confirmed
• Evaluation split origin: DAPFAM ALL title-abstract-claims to title-abstract-claims patent-family retrieval
• Train/eval overlap audit: not_audited
• Leakage note: exclude NanoDAPFAM evaluation query and target families and citation labels
• Multi-positive training: citation_family_multi_positive
• Useful training data: patent claim retrieval, family-level citation prediction, prior-art search pairs with claim text