MNanoBEIR / NanoBEIR-vi / NanoDBPedia

Overview

NanoDBPedia in the Vietnamese NanoBEIR slice is an entity-oriented retrieval task derived from DBpedia Entity Retrieval. The queries are Vietnamese translated entity information needs, and the corpus contains Vietnamese translated DBpedia-style entity descriptions. The task asks a model to rank entities that satisfy short needs involving names, categories, places, works, organizations, or entity classes. It is a compact diagnostic for Vietnamese entity search, alias handling, and type-aware retrieval.

Details

What the Original Data Measures

DBpedia Entity Retrieval evaluates entity ranking for information needs over a structured encyclopedic collection. In BEIR-style retrieval, entity descriptions are treated as documents. Relevance can depend on names, aliases, types, places, categories, and relationships. A relevant entity must satisfy the intent of the query, not simply share a visible token.

The Vietnamese translated version tests short-query entity matching with translated descriptions and mixed entity names. Some names remain English, while surrounding descriptions are Vietnamese. A strong retriever must combine exact name matching with semantic type and category understanding.

Observed Data Profile

The task contains 50 queries, 6,045 documents, and 1,158 relevance judgments. It is strongly multi-positive, with an average of 23.16 positives per query. The minimum is 1, the median is 18.0, the maximum is 81, and 48 queries are multi-positive, or 96.0% of the query set. This makes the task a broad entity ranking benchmark.

Queries average 35.04 characters, while documents average 358.04 characters. Queries are short and often keyword-like, while the documents are concise entity summaries. Ranking must cover many acceptable entities for broad needs, not just the most obvious exact match.

BM25 Evaluation Profile

BM25 reaches nDCG@10 of 0.4949, hit@10 of 0.9000, and recall@100 of 0.6010 using the top-500 BM25 candidate subset. This is a strong lexical baseline. Entity queries often include names, locations, or category words that appear directly in the relevant descriptions.

The recall@100 value shows that exact overlap still misses a significant part of the relevant set. Entity needs can involve aliases, translated category labels, or broad classes of entities. BM25 can find obvious name matches, but it is less reliable for type-aware coverage across many positives.

Dense Evaluation Profile

The dense harrier-oss-270m run reaches nDCG@10 of 0.5676, hit@10 of 0.9600, and recall@100 of 0.7098. Dense retrieval is strongest across the main metrics. This indicates that embedding similarity captures type, category, and semantic entity constraints beyond surface words.

Dense retrieval is especially helpful for short entity needs that are underspecified or category-like. It can connect a query such as films shot in a city, former republics, or architecture in a location to entity descriptions that satisfy the type constraint without sharing all terms exactly. Remaining errors likely involve ambiguous entity classes or broad queries with many relevant results.

Reranking Hybrid Evaluation Profile

The reranking_hybrid candidate set reaches nDCG@10 of 0.5628, hit@10 of 0.9400, and recall@100 of 0.7029. It uses exactly 100 candidates per query, with no rank-101 safeguard rows. The hybrid profile is close to dense retrieval but slightly weaker on all three metrics.

This suggests that lexical evidence is useful, but dense ordering is already well aligned with the Vietnamese entity retrieval task. Hybrid search remains a strong candidate source, especially for exact names, but it does not improve over dense retrieval in this slice. The final ranking should keep semantic type matching central.

Metric Interpretation for Model Researchers

Because almost every query has many positives, hit@10 alone is not enough. A model can find one entity early while missing many other relevant entities. nDCG@10 measures first-page ranking quality, while recall@100 measures relevant-set coverage for downstream reranking or browsing.

The method comparison shows that BM25 is strong for exact entity anchors, dense retrieval is best for broad entity ranking, and reranking_hybrid is competitive but not superior. NanoDBPedia-vi is useful for testing whether models can combine entity names with type and category constraints.

Query and Relevance Type Tendencies

Queries include entity needs such as a Fitzgerald Auto Mall in Chambersburg, a 1994 Alice Munro short-story collection, Gallo-Roman architecture in Paris, former Yugoslav republics, and films shot in Venice. Relevant documents are DBpedia-style descriptions of entities satisfying those needs.

The task rewards constraint tracking. A query may specify a work type, location, author, historical group, or class of films. A same-name or same-location entity can be wrong if it violates the intended type.

Representative Failure Modes

Likely failures include retrieving the wrong entity type, over-ranking famous entities that share a name fragment, missing aliases or translated category labels, and failing to cover broad entity classes. BM25 may be too literal, while dense retrieval may blur related entity types. Hybrid systems can inherit both errors if lexical anchors overpower type semantics.

Training Data That May Help

Useful training data includes Vietnamese entity search, Wikipedia or DBpedia retrieval, alias matching, multilingual entity linking, type-aware ranking, and hard negatives sharing a location, name, category, or entity type while violating a key constraint.

Model Improvement Notes

A model targeting this task should combine exact name precision with semantic type awareness. Sparse systems need alias and morphology handling. Dense systems should improve constraint tracking for short queries. Hybrid systems should preserve exact entity anchors without weakening the dense model's ability to rank broad relevant sets.

Example Data

Source Reference Table

Representative query and positive entity snippets:

Dataset Information

Candidate Subsets