NanoSSRB / LLMAgentAndTool

Overview#

LLMAgentAndTool evaluates retrieval over JSON records describing agents, tools, capabilities, model outputs, API requests, sessions, security events, and tool usage. Its 200 English queries search 10,000 documents with 600 positive qrels. Requests combine exact endpoints, model names, users, thresholds, status codes, or runtime settings with semantic capabilities such as summarization, sentiment analysis, or safe tool choice. The task measures whether a retriever understands software-like constraints without reducing records to bag-of-words matches.

Details#

What the Original Data Measures#

The SSRB formulation targets complex querying across heterogeneous schemas. This domain connects that problem to agentic and RAG systems, where retrieving the wrong tool or execution record can cause an incorrect downstream action.

Observed Data Profile#

Queries average 362 characters and documents 590 characters, with three positives per query. Examples use model names, confidence ranges, temperature, endpoint latency, response size, user identity, and capability descriptions.

BM25 Evaluation Profile#

BM25 reaches 0.3582 nDCG@10, 0.6700 hit@10, and 0.8617 Recall@100. Endpoint, model, parameter, and capability tokens are powerful anchors, but lexical search cannot consistently enforce ranges or infer equivalent capabilities.

Dense Evaluation Profile#

Dense retrieval reaches 0.4171 nDCG@10, 0.7300 hit@10, and 0.8367 Recall@100. It improves top-rank semantic matching but loses some exact-token coverage, especially for identifiers and configuration values.

Reranking Hybrid Evaluation Profile#

Hybrid is strongest at 0.4727 nDCG@10, 0.7750 hit@10, and 0.9067 Recall@100. The complementary gain indicates that tool and agent retrieval needs both exact software anchors and semantic capability understanding.

Metric Interpretation for Model Researchers#

High Recall@100 matters for tool availability; nDCG@10 matters when an agent will act on the first few results. Hybrid gains are evidence that endpoint and model tokens should not be discarded. Inspect whether errors violate runtime bounds, choose the wrong capability, or confuse a request record with a tool definition.

Query and Relevance Type Tendencies#

Queries cover tool discovery, execution-log search, generation settings, latency and size filtering, model-output analysis, access control, and security events. Relevance requires the correct record type plus all requested values.

Representative Failure Modes#

Common mistakes include selecting a semantically suitable tool with the wrong endpoint, ignoring latency or token limits, conflating confidence with sentiment score, or matching the right user in the wrong session. Version and case normalization can also merge distinct software identifiers.

Agent-and-Tool Notes#

Preserve paths, model IDs, parameter names, status codes, timestamps, and units. Semantic similarity should bridge capability paraphrases while keeping operational fields distinct; /summarize is not interchangeable with a record that merely mentions summaries.

Training and Leakage Notes#

Exclude NanoSSRB test records and disclose Struct-IR exposure. Avoid generating training pairs from evaluation tool definitions or logs, even with superficial renaming.

Model Improvement Hints#

Use schema-aware serialization, identifier-preserving tokenization, range-aware features, and negatives with one wrong endpoint, model, user, status, or runtime value. Capability-aware reranking should verify every constraint.

Training Data That May Help#

Independent API catalogs, tool-use traces, developer documentation retrieval, and synthetic execution logs can teach this behavior without benchmark overlap.

Synthetic Data Guidance#

Generate fictional tools and execution records with grounded natural-language requests. Include close alternatives differing in endpoint, permission, latency, model, or parameter; do not seed from evaluation data.

Source Reference Table#

Example Data#

Dataset Information#

Candidate Subsets#