Microblog

There's a lot of stuff I love about the Agent Score directory that Fern put together, but in particular I think their quotes from industry leaders say better than I've been saying for the last month how wrong companies are that have been downsizing their docs teams.

This is not theoretical. These are real companies that have figured out how to measure the impact of AI agents in their conversion pipeline, and have committed to agent experience as a key business differentiator. And have connected that throughline directly to agent-readable documentation.

These folks are ahead of the curve, and their product adoption will show it. Companies like Snowflake that have dropped their entire docs team are going to see the results of these decisions and not have the foggiest idea how to measure or attribute it.

Don't be like Snowflake. Learn from the folks who are leading the pack here and figure out how to measure and apply these principles in your business.

AX is the thing.

Links from the comments:

• https://buildwithfern.com/agent-score

Another busy weekend for the AFDocs tool! I released v0.9.0 with some additional --verbosity details on page-level checks, and support for running with a config from the command line. Define the checks you want to run in a config and easily re-run as needed while you're fixing up specific issues.

I also put together a real docs site for it, with details about the new scorecard format, how to use the CLI, using it in CI, and using the library's programmatic access. And if you're a docs team wondering what to do with all this, there's an "Improve Your Score" guide, and comprehensive per-check breakdowns to walk you through what each check means and how you can fix them.

https://afdocs.dev/

When I started building skill-validator, I didn't know I was starting a second full-time job as an AI researcher and tool builder. But here we are! My little project of researching what makes a good-quality Agent Skill has turned into two tools, skill-validator and afdocs, 6 live websites + 1 WIP, and a bunch of GitHub repos backing it all.

For my weekend project, I decided to build an analytics dashboard to keep an eye on activity as my projects grow. And since I'm all-in on the AI/agent space, I wanted to elevate AI and agent traffic to first-class consumer categories alongside human viewers.

This is all totally custom logic, with no analytics platform involved. No cookies, either, so I don't have to display any cookie warnings and my analytics aren't contributing to people getting tracked across the web. I'm just pulling my server logs and parsing them to collect analytics I care about.

I'll feed my findings here back into what I'm learning about how AI and agents use the web. This gives me another view into the whole problem domain. And now I'll have my own data points to help me evaluate the claims that services are making about what their tools are capable of, and how they're seeing AI consume web content.

(Spoiler alert, I've already been keeping an eye on platforms that are incorrectly reporting/attributing things they're seeing in their data... so now I'm going to have some data of my own to tell you why they're wrong 😉 )

Instead of shoving AI into all the software that doesn't need it, can someone please build the collaboration tool that is missing from my workflows?

I live in GitHub, the IDE, and the terminal all day. I use AI regularly and my local files are all markdown for simplicity.

I am increasingly working with stakeholders who live in meetings all day and rely on Google docs. They need to dip in and out of content, want it to be nicely formatted so they can scan it quickly, and want a good mechanism to have discussion about specific elements.

Sure, GitHub can do all that, but my stakeholders aren't there and it doesn't fit their flows.

So instead, I'm saving Google docs as markdown, opening them in my IDE, working with them there, then copying the contents, going back to the Google doc, and pasting as markdown to update the doc.

Can someone please just make something that can sync content between Google docs and markdown on my local file system so I don't have to do this stupid dance? Ideally, it could also sync comments so I don't have to copy/paste them, too.

I would pay money for this. Enterprises would probably pay a lot of money for this.

(Editing to add, based on a lot of comments about switching to other platforms - this isn't a "switch to another platform" situation. It's a "my enterprise uses Google docs for internal collaboration" situation, and trying to get things into and out of that moat for use with my agents.)

Do your docs fill any customer support obligations? Do you have SLAs or customer agreements that require docs to be available to your customers?

Well, here's a Thursday afternoon thought exercise for you... with more and more software development being outsourced to agents, when do "agent-friendly docs" become an SLA issue? When does it start showing up in customer agreements?

I think we're very early in growing awareness of this being it's own thing to measure, but I'm really curious what y'all think about this.

(Also, good reminder you can use `npx afdocs check https://your.docs.domain.example.com to check whether your docs are agent friendly, and start thinking about what you need to do to get there.)

Oh noes. I just keep making work for myself. 😅

I performed an analysis of my PR review feedback (over 100 comments across 7 PRs) for the recent Agent Skills project I worked on.

Now I’ve created 12! new issues on the ‘skill-validator’ repo for checks to automate or semi-automate detection of specific patterns of issues I commented on repeatedly.

Six are structural checks:

• Platform-specific tool names in the skill (not cross-platform portable)
• Shell-specific commands without multi-shell handling
• Flag/version/preview/beta features without stability notes
• Near-duplicate text blocks within a skill
• More specific token budget analysis within a skill to help authors be more efficient
• Requiring eval results when an eval testing directory is present

And six are LLM-as-judge scoring candidates:

• Decision tree completeness scoring
• Vague instruction detection
• Reference file routing quality
• Flagging external links and abstract verbs without actionable next steps
• Cross-file consistency checking
• Detecting conditional content in SKILL.md that belongs in reference files

Once I get these checks implemented, passing the ‘skill-validator’ quality threshold will get harder. And I bet some people will be unhappy with that because skills that are passing in CI will start failing. But these all relate to real instructional quality and other quality issues that will negatively impact agent outputs.

In for a penny, in for a pound! If I can automate some of this stuff successfully, it will reduce the human review burden and improve skill efficacy for teams that don’t have a “me” in the loop 😉

The Agent-Friendly Docs Spec just got a spec bump! In v0.3.0, the spec:

• Folds the standalone llms-txt-directive from the former "Agent Discoverability Directives" category into the llms.txt category, and renames it "Content Discoverability"
• Adds a new "Recommended action" to each check with details about what to do to resolve a warn or failure state
• Adds a new "Interaction Effects" section detailing the ways certain check failures can combine to create disproportionately worse outcomes for agents

Working on updates to afdocs now to incorporate spec updates, and I expect those to land in the next couple of days.

Excited to keep building in this space, and keep adding details to the spec as we discover more about agent docs consumption patterns. 🎉

https://agentdocsspec.com/

Really excited to celebrate the release of the official MongoDB Agent Skills 🎉

I had the opportunity to work with an excellent cross-functional team of collaborators to bring this initial collection of skills to your favorite coding agent.

This project has been a major proving ground for everything I've been posting about Skill development and quality. From day 1, we had CI running a structural and qualitative validator, we developed a custom review skill to help authors self-evaluate, and we had rigorous human review for technical accuracy, instructional quality, actionability, and consistency. Our contributors iterated through hundreds of review comments and multiple rounds of testing to produce this curated collection of skills.

Appreciative of all the work that all of our skill authors and contributors put in to bring you a high-quality collection! Now we get to move on to the next phase: figuring out how to operationalize contributions at this high quality bar on an ongoing basis 😁

If you give them a try, I'm curious to hear your feedback!

Sharing: MongoDB's announcement: Introducing MongoDB Agent Skills and Plugins

Thanks to a contribution from @marc0olo, the skill-validator tool can now use Claude Code in headless mode to do LLM-as-judge scoring! 🎉

Before this change, you could perform structural validation fast and free, but the qualitative LLM-as-judge scoring to assess your skill across multiple dimensions was only available with an Anthropic or OpenAI API key.

Now, if you've got a Claude Code subscription through work or your personal org, you can use your Claude CLI binary to perform the LLM-as-judge scoring.

One caveat: headless mode still loads your CLAUDE.md, skills, and other context, which can affect scoring output versus the bare API calls. But for folks who aren't as sensitive to the possible context pollution affecting scoring reproducibility, this feature makes LLM-as-judge scoring accessible without the extra cost of the API calls.

Available now in the v1.5.0 release.

And this is why I love open source 😁

Another busy weekend for afdocs. The tool is up to v0.7.1 with fixes and improvements:

• Many fixups for the markdown-content-parity check - turns out, comparing HTML and markdown is complicated! I'm sure there's more to do there.
• Several improvements for running the tool's checks in CI:
  • Implement the describeAgentDocsPerCheck vitest helper to get a detailed per-check breakdown in CI
  • Improvements for config resolution and default timeouts
  • A copyable example setup showing you exactly what files you need to set up the tests and CI in your project

I now have my personal website and my 4 websites for agent-related stuff all fully passing the agent-friendly docs checks! 🎉 And CI is set up to run the checks on every push or PR to main for every site to make sure they stay that way as I continue to add content.

I also did find that the markdown-content-parity check helped me identify a lot of missing content from my markdown pages. So now the markdown templates I'm using for Hugo are much more complete. 🎉

And anecdotally, while Claude was trying to identify the missing content to fix up some of my pages, it kept attempting to fetch the HTML versions of the page and being served the markdown due to content negotiation 😁 It's exactly what you want in regular usage, because the markdown is much cleaner for agents, but I had to keep telling it to curl the pages instead of fetch them as a result 😉