Research note · Adjacent literature
Adjacent literature: skills, plugins, memory, and coding-agent adaptation
A primary-source review connecting direct Skill benchmarks with procedural memory, retrieval, adaptation, and coding-agent research.
Research note prepared 2026-08-31. This is an adjacent-literature review, not the central review of the named personalized-skills paper.
Scope and reading standard
Here, “skill” is used as an operational umbrella for a reusable instruction, procedure, executable program, tool composition, or experience-derived memory that is supplied to an agent at inference time or accumulated across tasks. “Plugin” is used more narrowly for an extension that exposes tools/APIs or an agent interface. These are related but not interchangeable: a tool call can be correct without being a reusable skill, and a skill can be useful without changing model weights.
I prioritized primary papers and their arXiv/conference pages. The 2025–2026 skill and memory papers below are preprints unless a venue is explicitly stated; their numerical claims should be treated as provisional until independently reproduced. “Capability evidence” means that a system can perform the proposed operation under the paper’s setup. It is not evidence of durable developer productivity, safety, or general user benefit.
Top synthesis
- Reusable procedures can compound performance, but retrieval and fit are the gating variables. SkillsBench v4 provides the cleanest paired evidence: curated skills raise pass rate across its matched conditions, while the realistic-skills study finds that gains shrink toward no-skill baselines when the agent must search a large library and select imperfect matches. Voyager, ExpeL, Reflexion, SkillCraft, and the newer procedural-memory studies show the mechanism in bounded settings. A library should therefore be evaluated as a retrieval-and-execution system, not as a bag of prompt text.
- Abstraction beats raw history, but abstraction can over-specialize. The procedural-memory benchmark reports that LLM-generated procedural abstractions transfer across changed entities better than embedding retrieval of trajectories. The continual-learning study finds that abstract procedures transfer more reliably than detailed trajectories, while negative transfer disproportionately hurts hard tasks. AFTER likewise reports broad and role-specific skills; cross-role generalization is not automatic.
- Tool/API research establishes a plugin capability, not personalization. Toolformer, Gorilla, and ToolLLM show that models can learn when/how to call APIs, retrieve API documentation, and compose multiple calls. Their datasets are mostly curated or machine-generated and evaluate API correctness on fixed catalogs. They do not establish that a plugin learns a particular developer’s conventions, remains reliable as tools change, or improves end-to-end engineering outcomes.
- Coding evidence points to context/interface design as a major causal lever. RepoCoder shows iterative repository retrieval improves completion; SWE-agent shows an agent-computer interface can materially change issue-resolution results. Both are useful analogies for project-specific skills, but neither is a longitudinal study of developer interaction histories. Repository benchmarks also conflate model, prompt, retrieval, interface, and execution effects.
- Cost, stale knowledge, and harmful transfer belong in the primary metric set. SkillCraft reports large token savings in its skill-reuse protocol, while the realistic-skills work reports that skill benefits are fragile. Memory papers identify retrieval competition/forgetting; the practical implication is to log success, cost, latency, stale-skill rate, and negative-transfer rate—not just pass rate.
Evidence matrix
| Source | Experimental object / task | Main reported result | Relevance to Skills/plugins | Important caveat / evidence status |
|---|---|---|---|---|
| Li et al., “SkillsBench” v4 (2026) | 87 tasks across eight domains with curated Skills and deterministic verifiers; matched no-Skills/curated-Skills runs across 18 model-harness configurations | v4 (14 Jun 2026) reports average pass rate 33.9% → 50.5% (+16.6 pp; 25.5% normalized); configuration gains +4.1 to +25.7 pp; ≤3 focused modules beat larger/exhaustive bundles | Strongest paired benchmark evidence that curated skills can help; also tests bundle size and smaller-model compensation | arXiv preprint; skills are curated and task-paired, so this is capability/efficacy evidence rather than realistic discovery or personalization; model-harness details and deterministic verifiers constrain external validity |
| Liu et al., “How Well Do Agentic Skills Work in the Wild” (2026, v1) | Agents retrieve from 34k real-world skills under progressively realistic settings; also Terminal-Bench 2.0 | Skill gains degrade toward no-skill baselines as retrieval/selection becomes realistic; query-specific refinement recovers performance; Claude Opus 4.6 pass rate 57.7% → 65.5% in their Terminal-Bench experiment | Closest direct test of a large, imperfect skill library and the selection/refinement loop | arXiv preprint (v1, 2026-04-06); “realistic” library/task construction and model version limit external validity; no human developer study; headline gain is setting-specific |
| Chen et al., “SkillCraft” (2026, v2) | 126 compositional tool-use tasks, 21 families, six domains; agents compose, save, cache, and reuse skills | Skill saving/reuse cuts token use by up to 80%; test-time tool-composition ability correlates with success | Direct benchmark for higher-level tool compositions and reusable skill libraries | arXiv/OpenReview-era preprint (v2, 2026-03-10); benchmark tasks and APIs are constructed/adapted, not workplace histories; “up to” is a best-case bound; correlation is not causation |
| Belikova et al., “Managing Procedural Memory…” / AFTER (2026) | 382 enterprise-style tasks, six roles, 22 procedural skills; local, cross-task, cross-role, and cross-model transfer | One refinement round improves aggregate performance 3.7–6.7 points; skills evolved from diverse multi-model traces reach 73.1% cross-model test accuracy | Tests transfer and skill evolution closer to production-role scenarios | arXiv preprint; realism is benchmark-defined; aggregate gains hide role/task variation; cost, stale-skill handling, and independent replication are not established |
| Hu, Long & Wang, “When Continual Learning Moves to Memory” (2026) | Sequential-task ALFWorld and BabyAI experiments; varies memory representation and retrieval organization | Abstract procedural memories transfer more reliably than detailed trajectories; negative transfer harms hard cases; finer organization can trade forward transfer for forgetting | Direct warning that an external skill/memory layer inherits continual-learning problems | arXiv preprint; embodied synthetic environments, not coding or user preferences; abstract’s results do not quantify production-scale behavior |
| Kohar & Krishnan, “A Benchmark for Procedural Memory Retrieval…” (2025) | ALFWorld dual trajectory corpora; isolates retrieval and tests six methods on stratified familiar/novel queries | Embedding retrieval works on familiar contexts but has a generalization cliff; LLM procedural abstractions transfer across changed contexts; corpus scale matters more than richer embeddings | Gives a diagnostic target for skill retrieval independent of task execution | arXiv preprint; ALFWorld only and retrieval-isolation setup; transfer to code repositories and real developer language is untested |
| Liu et al., “Harnessing LLM Agents with Skill Programs” / HASP (2026) | Web-search, math-reasoning, and coding agents; skills become executable Program Functions that activate/intervene | Inference-time PFs improve web-search results by 25% vs multi-loop ReAct in the reported setup; post-training/controlled evolution reports 30.4% over Search-R1 | Tests executable guardrails rather than passive skill text and makes activation a first-class design problem | arXiv preprint; heterogeneous baselines and teacher-reviewed/evolved artifacts add confounds; PFs are not the same as user-specific skills; replication pending |
| Wang et al., “Voyager” (2023, v2) | GPT-4 agent explores Minecraft with automatic curriculum, executable skill library, and environment-feedback prompting | Reports 3.3× more unique items, 2.3× travel distance, and up to 15.3× faster tech-tree milestones than prior systems; reuses library in a new world | Foundational demonstration that executable, compositional skills can compound without weight updates | Latest arXiv v2 (2023-10-19); one game, one agent/model setup, and open-ended exploration metrics; not a controlled developer-personalization trial; benchmark transfer is narrow |
| Zhao et al., “ExpeL” (AAAI 2024) | ReAct-style agents gather trajectories and extract natural-language insights; evaluated on decision tasks including HotpotQA, ALFWorld, and WebShop | Reports consistent performance improvement as experience accumulates and explores transfer | Early template for converting traces into reusable textual lessons without parameter updates | Task-specific training-task accumulation, model/API choices, and self-generated insights can confound gains; no coding or longitudinal user study; abstract does not give a single cross-task effect size |
| Shinn et al., “Reflexion” (NeurIPS 2023, v4) | Agents turn scalar/environment feedback into verbal reflections stored in bounded episodic memory; sequential decision, reasoning, and code tasks | Reports 91% HumanEval pass@1 vs 80% GPT-4 baseline in its setup, plus gains on ALFWorld/HotpotQA | Establishes verbal feedback as a reusable episodic artifact and a precursor to skill distillation | Latest arXiv v4 (2023-10-10); multiple trials and test/compiler feedback are part of the method; comparisons are prompt/model/budget-sensitive; reflections are not necessarily executable or portable across users/projects |
| Schick et al., “Toolformer” (2023) | Self-supervised fine-tuning to decide when/how to call calculator, QA, search, translation, and calendar APIs | Improves zero-shot downstream performance and often competes with larger no-tool models, using only a handful of demonstrations per API | Foundational evidence for learning tool invocation policy rather than merely exposing a plugin | Fixed APIs and demonstrations; requires model training; does not study tool discovery, user-specific preferences, API drift, or end-to-end agent workflows |
| Patil et al., “Gorilla” (2023) | Fine-tuned LLaMA-family model generates calls for HuggingFace/TorchHub/TensorHub APIs; optional documentation retriever | Reports better API-call writing than GPT-4 on APIBench and less hallucination; retrieval adapts to changed documentation | Shows that a plugin can be paired with retrieval to follow versioned API docs | APIBench is a narrow curated catalog; “surpasses GPT-4” is benchmark/model/version-specific; API-call correctness is not tool outcome correctness or user benefit |
| Qin et al., “ToolLLM” (ICLR 2024, v2) | ToolBench: 16,464 RapidAPI REST APIs across 49 categories; ChatGPT-generated instructions/paths; ToolLLaMA with retriever and depth-first search | Reports complex instruction execution, unseen-API generalization, and performance comparable to ChatGPT; evaluates with ToolEval | Large-scale evidence for retrieval plus multi-tool planning, useful for plugin discovery/composition | Latest arXiv v2 (2023-10-03); data and solution paths are largely machine-generated; RapidAPI availability and evaluator assumptions limit realism; trained/evaluated catalogs can leak structure; reliability, security, and cost are secondary |
| Zhang et al., “RepoCoder” (EMNLP 2023) | RepoEval repository-level line, API-invocation, and function-body completion; iterative similarity retrieval + generation | Improves in-file completion by >10% in all reported settings and beats one-shot repository RAG | Strong analogue for project-specific context retrieval and iterative refinement | Completion is not autonomous issue resolution; offline benchmark and repository overlap/contamination are concerns; no learned developer profile or longitudinal adaptation |
| Yang et al., “SWE-agent” (NeurIPS 2024, v3) | Custom agent-computer interface for navigating repositories, editing files, running tests; SWE-bench and HumanEvalFix | Reports 12.5% SWE-bench and 87.7% HumanEvalFix pass@1 and finds ACI design changes behavior/performance | Evidence that the “plugin surface” (commands, navigation, feedback) can be causal, not just model weights | Latest arXiv v3 (2024-11-11); results depend on GPT-4 Turbo, ACI/prompt design, and benchmark version; pass@1 is not developer productivity; no personalization or multi-session skill reuse |
Source notes and implications for an agent-Skills program
1. Direct skill-library and procedural-memory evidence
Li et al. (2026), SkillsBench v4. The latest arXiv record is v4, revised 2026-06-14 (not the older v1/v2 numbers). Its paired design compares the same 87 tasks with and without curated skills under 18 model-harness configurations, reporting 33.9% → 50.5% average pass rate (+16.6 points), with +4.1–25.7-point configuration-level gains. The result is unusually useful because the no-skill condition is matched and verifiers are deterministic; the paper also reports that focused bundles of at most three modules beat larger/exhaustive bundles. This supports a focused-skill and matched-ablation design. It does not answer whether an agent can discover the right skill from a large library or whether developer-specific skills help: skills are curated and paired to tasks.
Liu et al. (2026), How Well Do Agentic Skills Work in the Wild. This is the most directly relevant stress test for a skills product: the agent must find a useful artifact in a 34k-skill collection rather than receive a hand-selected skill. The paper’s central empirical claim is conditional: skills help when relevance and quality are adequate, but the benefit erodes under realistic retrieval and selection. Query-specific refinement is presented as a recovery mechanism, including the reported Terminal-Bench 2.0 improvement. For a product, this supports measuring retrieval recall, selection precision, refinement success, and no-skill deltas separately. The preprint does not establish that the same selection dynamics hold for developer-authored skills or personalized coding traces.
Chen et al. (2026), SkillCraft. SkillCraft operationalizes a skill as a higher-level composition of atomic tools that can be saved and reused. Its contribution is primarily a benchmark/protocol: repeated structure and cross-task reuse become measurable. The reported 80% token reduction is evidence of efficiency under its “Skill Mode,” not a general claim that every skill library reduces cost. The most useful design lesson is to evaluate cold-start skill acquisition and warm-start reuse independently, with success and token/wall-clock cost reported together.
Belikova et al. (2026), AFTER. AFTER is notable because it explicitly tests local improvement and transfer across tasks, roles, and model backbones. The reported 3.7–6.7-point refinement gains and 73.1% cross-model accuracy from diverse traces suggest that provenance diversity may improve portability. At the same time, the paper reports some skills becoming role-specialized. A real system should retain scope metadata and reject cross-role reuse by default when evidence is weak.
Hu, Long & Wang (2026). This study reframes continual learning as a memory-management problem. Its key warning for Skills is that adding a persistent artifact does not eliminate forgetting: retrieval competition, representation, and organization decide what is actually available at inference. “More granular” indexing is not unconditionally better. This supports bounded, auditable libraries with explicit replacement, staleness, and rollback policies.
Kohar & Krishnan (2025). By isolating retrieval from execution, this benchmark helps prevent a common evaluation mistake: treating a successful downstream run as proof that retrieval understood a procedure. Its generalization cliff implies that lexical/vector similarity may be inadequate when a developer renames files, changes frameworks, or expresses the same intent in unfamiliar language. Procedural summaries should encode preconditions, ordered steps, expected observations, and failure recovery—not only keywords.
Liu et al. (2026), HASP. HASP turns a passive instruction into a program with activation and intervention hooks. This is relevant to plugins that need to enforce a sequence, check a state, or correct a repeated failure. However, its claims mix inference-time intervention, post-training, and controlled evolution; those should be treated as different interventions in any replication. It is evidence for an executable skill architecture, not evidence that automatic skill evolution is safe.
2. Memory and tool-use foundations
Voyager, ExpeL, and Reflexion form a useful progression: executable code skills in an embodied environment; natural-language insights extracted from trajectories; and verbal reflections retained as episodic memory. Together they support the mechanism “observe outcome → summarize reusable procedure/lesson → retrieve it on a later attempt.” They also expose portability limits: Minecraft behaviors, ALFWorld lessons, and HumanEval reflections are not automatically project-specific coding skills. In particular, benchmark feedback (compiler/tests/environment rewards) is much cleaner than the sparse, ambiguous feedback in developer interaction histories.
Toolformer, Gorilla, and ToolLLM establish complementary plugin capabilities: learn an invocation policy, ground calls in changing documentation, and retrieve/compose among thousands of APIs. Their strongest evidence is about syntactic/API-use competence on fixed or constructed catalogs. A production plugin layer still needs authorization boundaries, version and schema checks, outcome validation, latency/cost accounting, and defenses against malicious tool descriptions. None of these papers should be read as evidence that a model has learned a developer’s preferences.
3. Coding-agent analogues
RepoCoder demonstrates that repository context is useful when retrieval is iterative: the model’s first completion can improve the next retrieval query. This is a plausible implementation pattern for project skills, but the paper’s task is code completion rather than issue resolution or developer collaboration.
SWE-agent demonstrates that the interface through which an agent reads, edits, and tests code changes outcomes. For Skills, this suggests evaluating not only the text of a skill but also the available commands, observation format, test feedback, and action budget. Its benchmark score is capability evidence under a specified ACI, not a causal estimate of productivity or maintainability.
Practical evaluation hypotheses suggested by the literature
- Retrieval: Compare no-skill, oracle-skill, library-retrieval, and retrieval-plus-refinement conditions. Report retrieval precision/recall and the fraction of runs harmed by a retrieved skill.
- Transfer: Hold out projects, frameworks, role vocabulary, and model backbones. Test both renamed-equivalent tasks and genuinely novel tasks; measure cross-task, cross-role, and cross-model transfer separately.
- Quality and safety: Track pass rate, regressions, token use, wall time, tool-call count, stale-artifact rate, and negative-transfer rate. Require provenance, scope, preconditions, and rollback for executable skills.
- Personalization: Use longitudinal, consented developer histories with a time split and project split. Evaluate whether the artifact improves the same developer’s task outcomes after controlling for model/prompt/interface changes, and include acceptance, edit distance, rework, and subjective burden—not only benchmark pass rate.
References
The links in the matrix are the canonical arXiv records (and identify the arXiv-issued DOI on each record). Version-sensitive results are taken from the latest record available on 2026-08-31; in particular, SkillsBench is v4 (revised 2026-06-14). Conference status is stated only where the paper’s record indicates acceptance: ExpeL (AAAI 2024), Reflexion (NeurIPS 2023), ToolLLM (ICLR 2024), RepoCoder (EMNLP 2023), and SWE-agent (NeurIPS 2024). All 2025–2026 direct-skills/memory entries are treated here as preprints.