Microblog

Another shoutout to @rhyannonjoy for contributing more known platform limits to the Agent Friendly Documentation Spec!

This time, Rhyannon dug into Cursor. Reading their writeup of the methodology reveals a lot of assumptions we're probably all making about how agents fetch data: https://rhyannonjoy.github.io/agent-ecosystem-testing/docs/anysphere-cursor/methodology

(And spoiler: we're probably wrong about at least some of them!)

And the friction log reveals how complicated it can be to test these things.

Super appreciative of their work here. Rhyannon has become a substantial contributor to the spec at this point with all the deep dives on platform-specific findings, and if you have been wondering how to test things like this, their site is a great learning resource.

Really grateful to have them along on this research journey!

If you're publishing Agent Skills for others to use: do you know what happens when your skill loads on a platform you haven't tested?

The Agent Skills spec defines a portable format. Write a SKILL.md once, users install it anywhere. Over 25 platforms support it. But "support" covers a wide range of actual behavior. Your skill's reference files might load automatically on one platform and be invisible on another. Your frontmatter might reach the model on one platform and get stripped on another. Your instructions might survive a long conversation on one platform and get silently pruned on another.

The spec's implementation guide says "every skills-compatible agent follows the same three-tier loading strategy." That guide was based on analysis of 7  platforms. At least 25 had shipped their implementations before it was published.

We don't actually know how most platforms handle skill loading. Nobody does. So help me find out!

I've started a community research project cataloging 22 loading behavior checks across 9 categories: loading timing, directory recognition, resource access, content presentation, lifecycle management, access control, structural edge cases, skill-to-skill invocation, and skill dependencies. The repo includes 16 benchmark skills with embedded canary phrases that reveal exactly what a platform loaded and when, without relying on model self reporting.

What we need now is people testing on real platforms. If you use Agent Skills on any platform (Claude Code, Cursor, GitHub Copilot, Roo Code, or any of the others), you can run the benchmark skills and submit your findings. Even partial results from a single platform are valuable. One empirical observation beats a hundred assumptions.

This is community science for the agent ecosystem. The findings will be published at https://agentskillimplementation.com and the full project (benchmark skills, test procedures, results template) is on GitHub: https://github.com/agent-ecosystem/agent-skill-implementation

If you're distributing official skills and care about them working across platforms, this is directly relevant to you. And if you've already noticed your skills behaving differently on different platforms, your observations are exactly what we're looking for.

I have been seeing posts from folks about Snowflake laying off its entire tech writing team. This is an absolutely bonkers decision for any org to be making today.

Your technical writing team is how you ensure that people can discover and learn your products. And the content flows across many systems.

It’s not just humans reading the docs, and hasn’t been for years. Model providers crawl docs to train their models on your products. Coding agents refer to your docs in-process to perform coding-related tasks for developers. Documentation is an essential resource for all of these systems.

Forward-thinking orgs understand this and have been building feedback loops to evaluate how well these systems understand products from the docs, and make targeted updates to docs to improve how the information flows across these systems. Docs are more important than they were when it was just humans reading them, not less. Humans could adapt to poor documention and sometimes succeed in spite of it. Today’s systems can not.

Any company that cuts its entire documentation team is telling me it does not understand how product discoverability and agent-first developer workflows work in today’s world. It is telling me that I should switch to a company that understands these things because my tools need high-quality, well-maintained documentation to succeed at the tasks I need to achieve.

Busy weekend for Agent-Friendly Docs!

• I added a new check to the spec, rendering-strategy, which checks whether a docs site renders content that is visible to tools like Claude Code and Cursor that use fetch library implementations instead of embedded or headless browsers
• I updated spec pass/warn/fail details based on practical details discovered during implementation
• I completed implementing the remaining checks in the afdocs CLI and library available through npm 🎉 All 22 checks are now fully implemented.

If you have Node.js/npm installed, you can run the tool locally to check your docs site for agent-friendliness:

npx afdocs check https://docs.example.com

The tool is now at v0.5.0; I'm going to leave it a v0.x tool for a bit longer while I wait to hear any bug reports from folks who are trying it out. Depending on how much feedback I get, and my own experience using it, I'm hoping to cut a 1.x release in April.

If you try it out, please do file any issues with bug reports or request enhancements on the tool repo. I'll throw links in the comments.

Links from the comments:

• https://github.com/agent-ecosystem/afdocs
• https://www.npmjs.com/package/afdocs
• https://agentdocsspec.com
• https://github.com/agent-ecosystem/agent-docs-spec

Two more checks implemented in v0.4.0 of the afdocs npm package! 🎉 This brings the tool up to 16 of 21 checks implemented.

If you haven't seen it yet, afdocs is a TypeScript CLI and library available through npm to help you check if your docs are agent-friendly. It implements checks against the Agent-Friendly Docs Spec, giving you immediate signal on where your docs are already agent-friendly, and where you could make enhancements.

The two new checks I've added in this version are:

• redirect-behavior: checks how redirects are implemented in your docs, and warns or errors if they're implemented in a way where agents may not be able to follow them
• llms-txt-directive: checks both your HTML and markdown pages to see if they have a hidden directive referring agents to an llms.txt file. This can help agents find agent-friendly content, like markdown instead of HTML, or serve as an index for agents to find related or relevant pages

If you've got Node.js/npm installed, you can run the tool on any docs site today with this command:

npx afdocs check https://docs.example.com

Just change the docs.example.com link to your docs site and give it a try! It samples random pages to give you info about how agent-friendly your docs are. You can specify how many pages to sample with the --max-links option; it defaults to sampling 50 pages.

You can also use it as a library to help define tests for agent-friendliness that you can run in CI.

I'll throw a link to the package on npm in the comments if you'd like to learn more.

Links from the comments:

• https://www.npmjs.com/package/afdocs

I’ve started reviewing some draft agent skills, and am learning a few things about optimizations we can make:

• If a skill has prerequisites that are implicitly stated somewhere in the skill body, move them to the top and explicitly state them as prerequisites. Provide a scope gate with instructions for how the agent should handle missing prerequisites. Should it prompt the user? Attempt to install something on its own? Take some local action to resolve the missing prerequisites?
• If a SKILL.md contains info that is only needed conditionally, consider moving it out of the skill body and into a reference file in ‘references/‘. Agents load the entire skill file into context every time it’s invoked, and if part of the content is not required for every invocation, you’re wasting context with information that only applies conditionally.
• If your skill relates to something that’s behind a feature flag, consider what happens if the feature flag is removed. Is the skill written in such a way that it’s still useful? Are there instructions for the agent about what to do once the feature flag is no longer there? Will it the skill need to be updated or removed, and do you have a way to let your users know when that needs to happen?
• Is your skill or any reference content tied to information that is maintained somewhere else? If so, changing the information in the other source should trigger a skill update. You need to have processes in place to watch external sources of truth and alert you when skills need to change.

If you’ve been creating or reviewing agent skills recently, any other optimizations you’ve spotted that we should be thinking about? 🤔

Another shout-out to @rhyannonjoy for contributing more known agent platform limits to the agent docs spec! Rhyannon has been systematically testing some of the other tools, and with their recent spelunking into Google Gemini, the picture has gotten even murkier for agent docs access.

Gemini works differently than some of the other agent platforms, and Rhyannon's observation here has implications for understanding how Gemini "sees" our docs:

There’s no direct equivalent to the Claude API web fetch track’s character-level truncation measurement. The API doesn’t expose the text injection or where truncation occurred.

The takeaway is that with the way the Gemini API works, there isn't a way for us to get clear visibility from the API side into what the agent retrieves, and where or if a truncation boundary may exist. It's not as simple or clear as knowing Claude Code or the Claude API truncates at X characters.

It's probably possible to do some observed testing; give Gemini URLs for pages with different lengths, and ask about content on different parts of the page to determine what Gemini "sees."

But the fact that there isn't a straightforward way to discover this information underscores the challenges we face as documentarians - what is true for one platform may not be true for others, so the question of "how well can agents use our docs" has a qualifier: "which agent?" For docs teams, the question in the short term may be "which agent platforms are most of our customers using?" and trying to optimize for that, instead of trying to generalize rules that work for all agents.

Anyway, big shout out to Rhyannon for continuing to explore this and sharing their findings back to the Agent-Friendly Docs Spec so we can all benefit! 🎉

I'm also really enjoying the write-ups of their research - I'll throw a link in the comments if you're curious about this stuff the way I am. 😁

Links from the comments:

• https://rhyannonjoy.github.io/agent-ecosystem-testing/

Agent Skills provide a great way to give your agent reusable workflows, information about up-to-date product details, or custom domain knowledge. But my audit of 673 agent skills last month found that 22% fail basic structural validation, and 52% of tokens across these skills were in non-standard files potentially wasting your agent's context.

In short, there's a real quality gap in published agent skills. And using skills that don't meet minimum quality standards can actually degrade your agent experience. So I wrote skill-validator to help people assess their skills.

As I've been socializing skill-validator with people creating skills, I realized some skill authors may need additional help interpreting the output or figuring out what changes to make to improve their skills. So I've added some examples of tools I've created to help authors ensure their skills meet a minimum quality bar to the skill-validator repo:

• A review-skill skill that instructs an agent how to run skill-validator, interpret the results, and provide actionable feedback to skill authors
• A CI workflow that runs structural validation checks on PRs with new or changed skills, and fails CI for skills that don't meet structural quality requirements, potentially blocking merge

Whether you're adding your own or someone else's skills to a project, or publishing official company skills for customers to use, these tools can help you ensure the skills you're consuming or producing meet minimum quality requirements.

Links from the comments:

• https://github.com/agent-ecosystem/skill-validator/tree/main/examples

If you like to take concrete actions to support causes, there's something you can do today to help women succeed: join Kiva, and fund micro-loans to help women grow businesses, bring much-needed community support programs to their communities, or make lives more sustainable for them and their families.

I've been a Kiva member since 2013, and here are some of my stats:

• I've helped fund 242 loans
• I've lent to 34 different countries and territories
• I've helped women fund agriculture, bakeries, convenience stores, grocery stores, computer services, and other much-needed services in their local communities
• I have lent almost exclusively to women, because as a woman, my goal with these micro-loans has been to help lift women up

When funds are repaid, you can re-lend them to keep the cycle going. So you can start with a small contribution and grow your impact over time. I usually re-lend when I've got $50-100 in repayments accumulated in my account, and I add another $50-100 of my own, which grows the impact I can have year-over-year without being unsustainable for me.

I'll throw a link in the comments to a special Kiva is running today - support a woman and get an extra $20 to lend.

Links from the comments:

• https://www.kiva.org/lend-by-category/women

Last weekend, I set up a Digital Ocean droplet to serve as a self-hosted runner for some GitHub CI I wanted to run in private repos in the new org I set up. I also set up Cloudflare Pages to host a site behind a GitHub App auth login so only people in my org can access it.

This weekend, I’ve added a FastAPI server to my Digital Ocean droplet to host a form. It’s got a Cloudflare tunnel to the site that’s behind GitHub App auth login, and I’ve slightly complicated deployment to make it more secure.

I do not have a DevOps background, but that doesn’t stop me with Claude’s assistance. BUT Claude’s DevOps guidance was not secure by default. At every step of the way, I’ve had to ask about possible attack vectors and how I can make the infra more secure. I’m sure I’ve missed things because I don’t have a DevOps background, even with asking about various security angles.

For my project, the exposure risk is pretty low. It’s really not the end of the world if anyone gets access to the data I’m working with. It’s all for internal/private use - I’m not working with customer data or selling a service.

But these projects have given me anxiety about all the new services folks are spinning up. I have to ask myself how many other people who don’t have DevOps backgrounds are now standing up infrastructure with the help of coding agents and are not asking the security-related questions. I admit, I’m probably going to hesitate now before signing up for someone else’s new service if it needs PII, payment info, or any data I don’t want exposed.

Also, this weekend Claude wrote a few additional details about me to its persistent memory… “prefers vi editor” (when I’m ssh-ing onto a remote server and the default editor is nano, I’m lost!) and “security-conscious.”

Ok, Claude, I think that’ll be helpful. 👍

Links from the comments:

• https://www.kiva.org/lend-by-category/women