Agent Skills Library

Research note · Ecosystem evidence

Ecosystem evidence: Skills, plugins, tools, connectors, and MCP

Definitions, evidence, and evaluation standards for separating procedural guidance from tools, connectors, protocols, and plugin packaging.

Research date: 2026-08-31 Scope: Evidence relevant to whether procedural Skills and packaged integrations improve AI-agent outcomes. Sources are linked to the paper, specification, or primary vendor documentation. Claims from preprints and vendors are labeled accordingly.

Working definitions

The terms are related but are not interchangeable:

MechanismWhat it isWhat it does not establish
SkillA portable, filesystem-based package: required SKILL.md (metadata plus natural-language procedural instructions), optionally with scripts, references, assets, examples, or templates. The Agent Skills specification describes progressive disclosure: load short metadata, then the body on activation, then resources as needed.A Skill is not automatically a new model capability or a permission boundary. Its instructions can be ignored, wrong, stale, or malicious; bundled scripts may execute with the host's permissions.
Tool/APIA callable operation with a contract/schema, such as a database query, code executor, browser action, or send_email. The model chooses when and how to call it; the host or runtime actually executes it.Tool access does not guarantee correct selection, arguments, sequencing, policy compliance, or safe use.
Connector/appA product integration that authenticates and mediates access to an external service, data source, or action. It normally carries account, scope, and workspace permissions.A connector's existence does not grant access beyond the user's/provider/workspace permissions, and does not itself improve task quality.
MCPAn open JSON-RPC protocol for host–client–server interoperability. Servers can expose resources, prompts, and tools; clients can support sampling, roots, and elicitation.MCP standardizes transport/discovery and capability exchange, not correctness, quality, trust, or security.
PluginA product-level package that groups Skills, apps/connectors, and sometimes app templates or MCP configuration for a workflow. It is primarily a discovery, distribution, and administration layer.Packaging does not identify which component caused an outcome. Plugin-level causal evidence is not yet available in the sources below.
Agent harness/runtimeThe loop that discovers Skills, injects context, selects/calls tools, manages state and approvals, and verifies results.“With a Skill/tool” is not a portable treatment unless the harness, model, context, permissions, and evaluation are held constant.

The open specification is explicit about the Skill boundary: SKILL.md contains instructions; scripts are optional executable code; the whole file is loaded after activation; and deeper references should be loaded on demand (Agent Skills specification). OpenAI’s current product documentation makes the same distinction: “Skills provide reusable instructions, while apps connect to external services,” and a plugin may contain either or both (Skills in ChatGPT, updated 2026-08-31; Plugins in ChatGPT and Codex).

Evidence table

Date / sourceMechanism and designMain claim or resultEvidence strength and caveat
2025-11-25, MCP specificationNormative protocol specification; defines hosts, clients, servers, resources, prompts, tools, and optional client features.MCP gives a common interface for connecting LLM applications to context and executable capabilities. The spec requires explicit consent and says tool annotations should be treated as untrusted unless from a trusted server.Definition/interoperability, not an efficacy study. MCP can lower integration friction, but it cannot make a tool correct or an agent reliable. Its own security section says protocol alone cannot enforce all safety principles.
2024-12-19, Anthropic, “Building effective agents”Practitioner synthesis from work with “dozens of teams,” contrasting workflows and dynamic agents; tools are part of an augmented LLM.Simple composable systems are usually preferable; agents trade latency/cost for task performance and can compound errors. Clear tool documentation and evaluation are central.Vendor experience/anecdotal synthesis, not a controlled comparison of Skills or plugins. Useful design hypothesis: add complexity only when evaluation shows it helps.
2025-09-11, Anthropic, “Writing effective tools for agents”Prototype tools/MCP servers, create realistic held-out evaluations, and iterate on namespacing, response shape, and tool descriptions.Anthropic reports that tool names/descriptions and high-signal responses materially affect tool selection and precision; it recommends a few narrowly scoped tools rather than a large overlapping set. It reports “dramatic improvements” after tool-description refinements on an internal SWE-bench evaluation.Vendor internal evaluation; useful but not independently reproduced. It supports tool/interface effects, not a general claim that every connector or MCP server improves outcomes.
2025-11-24, Anthropic, “Advanced tool use”Internal comparison of conventional loading versus deferred tool discovery (“Tool Search Tool”), programmatic tool calling, and examples.Anthropic reports 58 tools consuming about 55K tokens and up to 134K tokens observed before optimization. Deferred discovery reduced the illustrated context from about 77K to 8.7K tokens (85% reduction); internal MCP evaluations reported Opus 4 improving 49%→74% and Opus 4.5 79.5%→88.1%.Vendor-reported internal benchmark. Strong evidence that context/tool-library management can affect performance and cost, but no public protocol, task, or independent replication in the article; not evidence for Skills or plugin packaging per se.
2023-07-31, ToolLLM / ToolBench16,464 REST APIs across 49 categories; ChatGPT-generated instructions and solution paths; fine-tuned ToolLLaMA with API retrieval and ToolEval.ToolLLaMA reportedly generalized to unseen APIs and approached ChatGPT on the authors’ tool-use evaluations.Empirical tool-use study, but not a Skills/plugin ablation. Fine-tuning, retrieval, data construction, and evaluator all change together, so it shows that training for API use can help—not the marginal effect of merely attaching a connector at inference time.
2024-06-17, τ-benchSimulated user conversations plus domain-specific policy and API tools; success checked against final database state; introduces reliability metric pass^k.Even state-of-the-art function-calling agents (including GPT-4o) succeeded on under half of tasks; retail pass^8 was under 25%, showing substantial inconsistency over repeated trials.Controlled benchmark of tool-agent-user behavior. It has no matched “without tools” condition, so it does not estimate tool uplift; it is strong evidence that tool access alone is insufficient for reliable action.
2024-06-19 (NeurIPS 2024), AgentDojoDynamic benchmark with 97 realistic tasks, 629 security test cases, external tools, adaptive attacks, and defenses; measures both utility and utility under attack.Current agents fail many benign tasks and are vulnerable to some attacks; the framework is designed to evaluate the security–utility trade-off as attacks and defenses evolve.Peer-reviewed benchmark. It evaluates tool-integrated agents, not Skills/plugins directly. It supports treating safety as an outcome alongside task success.
2024-03-05 (Findings of ACL 2024), InjecAgent1,054 indirect-prompt-injection cases spanning 17 user tools and 62 attacker tools; evaluates 30 agents.ReAct-prompted GPT-4 was vulnerable in 24% of cases; reinforced attacker instructions nearly doubled its attack success rate.Empirical security benchmark. Attack rates depend on prompts, models, tools, and threat model; the result is not a universal rate. It does establish that external content plus tool authority creates a measurable failure mode.
2023-02-23, Greshake et al., indirect prompt injectionDemonstrations on real-world and synthetic LLM-integrated applications, including retrieved content and code-completion systems.Retrieved text can blur the boundary between data and instructions, manipulate functionality, control API calls, exfiltrate data, and in some settings induce arbitrary code execution.Foundational empirical security work. It predates MCP and does not quantify current clients, but the underlying data/instruction-confusion failure applies to Skills, tool outputs, and connectors.
2026-02-13 (v4: 2026-06-14), SkillsBench and HTMLMatched with/without curated-Skill conditions on 87 tasks across 8 domains and 18 model–harness configurations; deterministic verifiers and public trajectories. Also tests self-generated Skills on three stacks.Curated Skills raised average pass rate 33.9%→50.5% (+16.6 percentage points; configuration range +4.1 to +25.7 pp). Focused one-to-three-Skill bundles beat larger bundles. Self-generated Skills were below no-Skill baselines by 8.1–11.5 pp. 13/87 tasks had negative Skill deltas.Strongest direct paired evidence so far, but not a fully isolated causal experiment. Authors note that Skills add context, so gains may partly be “more context”; length-matched/random-text and retrieval-only controls are future work. Results are terminal/containerized and may not generalize to GUI, multi-agent, or long-horizon work.
2026-03-16, SWE-Skills-Bench and HTML49 public SWE Skills paired with fixed-commit GitHub projects and about 565 requirement-driven tasks; Claude Code + Claude Haiku 4.5; deterministic execution tests with and without the Skill.39/49 Skills produced zero pass-rate improvement; average gain was only +1.2%. Seven specialized Skills improved up to +30%; three degraded performance up to -10% when guidance conflicted with project versions. Token overhead ranged from savings to +451% with unchanged pass rates.Direct, SWE-specific paired evidence, explicitly labeled preliminary/work in progress. Skill selection and task construction are curated; one model/harness is used. It strongly cautions against blanket Skill adoption and highlights contextual compatibility.
2026-03-31, SkillReducerEmpirical analysis of 55,315 public Skills plus a debloating pipeline; studies routing descriptions, body content, and reference loading.26.4% lacked routing descriptions; the authors estimate over 60% of body content was non-actionable and that references can add tens of thousands of tokens. In a 600-Skill evaluation, the paper reports routing preservation for 536/536 non-obsolete Skills after selective restoration.Preprint and largely a quality/cost study, not a user-outcome trial. The data support token and routing risk; “non-actionable” labels and the optimization benchmark need independent replication.
2026-08-09, “What Keeps Agent Skills from Being Reusable?”Static audit of 138,133 public SKILL.md files from 20,556 repositories; two-tier defect taxonomy plus routing stress test on 20,000 Skills.91.8% had at least one detected defect (88.8–94.6% across thresholds); common issues were routing metadata, bloated/non-actionable bodies, and poor resource organization. Clean routing metadata was retrieved more reliably in the stress test.Large-scale preprint audit. It measures artifact quality and routing, not end-to-end task success; detectors and public-GitHub sampling can miss defects or overstate ecosystem prevalence.
2026-08-11, GitSkillsDataset of 3,797,117 SKILL.md occurrences from 282,200 public repositories (July 2026), grouped into 1,877,981 distinct contents, with repository and some history metadata.Skills are already a large, copied, decentralized software artifact population; selection is probabilistic and there is no compiler/type checker, central registry, or package manager.Ecosystem measurement, not efficacy evidence. Public repositories are a lower bound and may overrepresent highly shared or low-quality artifacts. It does, however, make provenance, versioning, and maintenance research possible.

Synthesis: what the evidence supports

  1. Curated procedural Skills can improve outcomes, but “Skills help” is not a universal effect. SkillsBench’s latest paired aggregate is positive and substantial (+16.6 pp), while SWE-Skills-Bench’s repository-grounded result is small (+1.2 percentage points on average) with 39/49 zero effects and some regressions. The most defensible conclusion is conditional: a Skill helps when it encodes specialized, actionable procedure that fits the task, model, harness, and project context.
  1. Packaging and personalization are different hypotheses. A plugin can make a good Skill/app easier to discover, install, authorize, and govern, but none of the reviewed evidence isolates “plugin packaging” as a causal treatment. Plugin-level outcomes should be decomposed into (a) procedural guidance, (b) tool/data access, (c) authentication/permissions, and (d) harness UX and routing.
  1. Tools and connectors add reach and ground truth, not automatic reliability. ToolLLM shows that models can be trained for broad API use; τ-bench shows that even capable function-calling agents remain inconsistent when policies, state changes, and user interaction matter. Tool descriptions, schemas, response quality, selection, and verification are first-class intervention points.
  1. More capability also means more attack surface. Indirect prompt injection and tool poisoning exploit the fact that tool results, retrieved documents, Skill text, and metadata can enter the same model context as instructions. A Skill that includes executable code or broad tool access should be treated like a software dependency with provenance, review, least privilege, sandboxing, version pinning, output validation, approvals, and audit logs.
  1. Context and routing costs are real. Anthropic reports tool-definition overhead of tens to more than 100K tokens in large libraries; SkillReducer and the reusability audit find widespread routing and content defects. Focused progressive disclosure is supported both by the specification and by benchmark/vendor evidence. More installed Skills or tools can mean more distractors, wrong selection, latency, and token cost.

References

← Back to the research synthesis