Microblog

How do we move forward as a documentation community to create agent-friendly docs that the new era of coding agents can consume and benefit from?

We make a spec! And we implement tools that can validate our docs against the spec.

I'm proposing the Agent-Friendly Documentation Spec - at the time of writing this post, it's 21 checks across 8 categories, but I keep adding new ones 😁

This is my attempt to turn observation and research into the minimum, practical, mechanical guidelines required to make docs agent friendly across these categories:

• llms.txt
• Markdown availability
• Page size
• Content structure
• URL stability
• Agent discoverability
• Observability
• Authentication

I'm sure there are more things I haven't considered yet, so help me brainstorm, technical writing hive mind... what else should be here? The website links to a GitHub repo. Please file issues and let's start talking about how this can serve all of us as we look to make our docs more coding-agent-friendly.

https://agentdocsspec.com/

Ok, lunch thought and I'm curious for your input, Technical Writers - no wrong answers here.

So far, as I've been digging into agent-friendly docs, I've been focusing mostly on static analysis - what can we measure - to derive some early findings and focus initial exploration.

But at lunch, I started thinking - what does "developer documentation" look like in 2 years? 3 years? 5 years?

Where do we draw the distinction in writing for developers - i.e. the agents - and writing for the humans running the agents? Are these distinct personas that require their own documentation sets? What belongs in each? And how do we serve the industries where regulation or other limits constrain their AI usage?

What do you think? Let's brainstorm!

I've done a deep dive analyzing the Agent Skills ecosystem around modular instruction packages that tools like Claude Code, GitHub Copilot, Cursor, and 27+ other agent platforms use. These packages are intended to extend what coding agents can do with custom instructions covering specific domain expertise.

I looked at 673 skills from 41 source repositories. Some things I found:

 - 22% of published skills fail structural validation — including skills from companies like Microsoft and Stripe  - Over half of all tokens contained in Skills are nonstandard files (LICENSE texts, XML schemas, benchmark data) that provide little or zero value to the agent  - Most skills just restate what the LLM already knows. My analysis indicates that novelty is the weakest quality dimension across the ecosystem, and the most important one. Novelty is information the model can't derive from training data.  - 66 skills look clean on the surface but carry hidden potential for contamination in their reference files

I also ran a behavioral eval to see whether structural risk scores actually predict degradation in agent output. They don't (r = 0.077). The real interference comes from content-specific mechanisms: template syntax bleeding into output, framework-specific patterns leaking into unrelated tasks, and API hallucination where the model invents plausible but nonexistent SDK methods.

I published an interactive report where you can explore every skill individually; validation status, quality scores, contamination risk, token breakdown; along with the full paper:

https://agentskillreport.com/

If you're writing or using Agent Skills, the practical takeaways:  - Run skill-validator before publishing  - Focus on what the LLM doesn't already know  - Clean nonstandard files out of your skill directory  - Scope skills to a single language SDK when possible

I'm working on a weekend project that requires an LLM to go read hundreds of docs links. And I am very dismayed by the state of docs for LLM consumption. I've been a little spoiled by being able to point LLMs at Markdown versions of docs pages for some of the bigger or more modern platforms, and it's a bummer how much software out there does not provide LLM-friendly documentation.

I keep falling back to GitHub repos, and specifically the raw versions of code and markdown files, and it works like a charm.

As an industry, if we can't figure out a way to elevate LLM/agent users to first-class consumers of our docs, we're just going to lose them entirely. "Skip the docs and refer to source" might become my first instruction instead of a fallback.

Oh noes.

I have set up an evaluation system to score LLM outputs for code gen tasks involving our products, which I’ve planned to use to validate any Agent Skill content we create.

For my first test case, the Skill content made the output worse than the unassisted LLM performance. 😬 🙃

We support 12 programming language drivers, plus a shell tool and some other interfaces.

Turns out, when you shove a code example in front of an LLM, it really wants to mimic that code example, even if it’s for a different programming language. So agents that can write the code perfectly fine from base training data actually mess it up when they see a “wrong” example in context.

For products that support a lot of languages… I’m wondering if this will require language-specific Skills? 😰 😱

Also, the code example patterns that we show in docs, which are written for humans to learn from - not intended to represent production code… yeah, maybe those are the wrong examples for LLMs. 😕

This is going to be an interesting learning curve…

Agent Skills are the new hot topic as AI-assisted engineering practices shift from prompt-focused to context-management.

To me, that means content governance and the content management lifecycle are back in vogue.

Nobody with experience managing content wants to maintain a duplicate source of truth. PostHog shared an article recently about an AI Wizard they created, and one of the problems they cited was drift between the maintained source of truth - docs and the product - and the AI Wizard content. Now picture that at scale across a bunch of different Agent Skills, most of which are duplicating content that already lives in your docs.

Some folks are solving for this by pointing LLMs at docs and saying “make this a Skill!”

Don’t do that.

Probabilistically copying your docs through an LLM is a recipe for omissions and conflation. And when you need to do it again in a month or 6 months when the product changes, you will get completely different outputs. It’s probabilistic.

Instead, beg, borrow, steal, or vibe-code your own system to deterministically create Skills from your docs source. That way, you can regenerate it as often as you need to as your product evolves, without maintaining a second source of truth or relying on a probabilistic system that will create unsustainable review and maintenance overhead.

Had a fun little project today: I put together a proof-of-concept for evaluating AI code gen for a task related to our product across various LLMs with a handful of inputs:

• Baseline: “basic” prompt only, no help
• With docs: same “basic” prompt, plus read in the content from a relevant documentation page
• With Skill: same “basic” prompt, plus read in the relevant content from a targeted AI Skill

I’m trying to validate whether:

• Unassisted LLMs can successfully perform code gen tasks for our product
• Adding a relevant documentation page produces an improvement in the output
• Adding a targeted Skill produces an improvement in the output, and if so, how that compares with the “With docs” version

The results were… unexpected! There’s definitely going to be a blog post about this soon.

One of the writers in the org just reported an issue with my tooling, which prompted me to look at our running GitHub Actions and I saw a sea of red.

Ran this little ghok command line utility (GitHub, are you OK?) and I can see there are active incidents affecting GitHub Actions.

Good reminder that having little utilities like this that meet you where you are make all the difference. I never have to look up the GitHub status page, or bookmark it, and wade through reports - just type a few characters into one of the many terminals I have open at any given time and see output like this.

If you are a technical writer who sometimes uses the command line for things, I cannot recommend enough starting to customize the utilities available to you to make your life a little easier. Before long, you will build up a library of useful things and wonder how you ever did anything without them.

Links from the comments:

• https://github.com/abennett/ghok

Heya #TechnicalWriting hive mind - I am curious for your thoughts!

I've been thinking a lot lately about the intersection of docs, AI skills, and custom agents as it relates to content reuse.

The name of the game in AI-assisted anything these days is context management. You don't want to blow up your context with an overly-verbose documentation page - or Skill file - and let's not even talk about handing an LLM an entire section of your docs.

I've been pondering how we can make our content more modular to enable just-in-time context discovery for LLMs. Sure seems to me like a new take on an old trick - DITA - but I'm not seeing anything like this implemented anywhere yet. Are you aware of any companies that have successfully modularized their docs content to support reuse in different AI contexts?

I've tried things like Skill Seekers to create Skills from docs content, but the verbosity is an issue. We really need much smaller chunks to right-size content to fit into a context window without overwhelming it, while human readers may have different preferences (fewer clicks, larger pages).

If you know of any company that is successfully doing something like this right now, please share!

Links from the comments:

• https://github.com/abennett/ghok

Yesterday was my 5-year anniversary at MongoDB 🎉

A lot has changed in 5 years. I was hired as the first non-developer on a team of developers documenting a DB that MongoDB had acquired. I learned Swift and released my first iOS app using our product. I learned C++ and wrote a TUI sample app for the product. Along the way, I embraced testing tools and drove IA initiatives that improved DX and feature discovery for our products.

After the product was sunsetted, my team pivoted to bringing programmatic standards, testing, and strategy to the larger org. On an average day, I’m far more likely to be writing code than docs, although I still do both. Internal tooling and enablement has become my happy place, all with the purpose of making our programmatic experience world class for both developers and robots.

Leadership at MongoDB talks about how they want working here to be an inflection point in people’s careers, and it has definitely been that for me. I’m doing the most interesting work I’ve done in my career, with a team of people who are a delight to work with.

I feel I’ve been blessed in this role, and I hope for many more years helping transform how my org - and the industry as a whole - handles programmatic user experience in docs.