Microblog

A thing I have been seeing is that it’s really hard for product teams to come up with realistic user prompts for running evals.

I have been leaning on traditional developer experience artifacts to help with that. If you’ve got personas, jobs-to-be-done, and user journeys for your product, you can pick a persona, a job, and potentially a journey stage to create realistic prompts for that user. So developer experience is still the right starting point for agent experience when it comes to writing eval prompts.

Another thing you can do is use public GitHub projects to derive eval prompts. If you’re tracking public projects where your product is used, you can look at PRs that have integrated or configured your product, and those are real user jobs to be done that you can derive prompts from. Public GitHub data carries a cheating risk, though, in that agents with Internet access can go look at the actual completed work on GitHub and provide an answer rather than reasoning about it.

If you run evals to understand how agents use your products, how do you derive prompts? Does anyone have any other good tips or techniques to share?

AFDocs had a busy weekend; two new releases for documentation teams who want to check their docs for agent-friendliness:

• v0.21.0 is a bug fix and non-breaking enhancement release; thank you to @JakeSCahill for reporting issues and @danielcalvolopez for a PR to emit progress notifications
• v0.22.x is a breaking release that introduces 5 new checks, bringing it up to spec v0.6.0, which changes scoring and may require sites to do new work to improve agent-friendliness

Notably, one of the new checks calls for markdown versions of pages to contain absolute links instead of relative. When migrating to v0.22.0, I had to update several of my sites that only contained relative links. This makes the markdown more portable, and solves several cases where agents could fail to construct the correct URL if it needs to load a related page.

Because v0.22.0 is a breaking change, I've added a new "Upgrading to v0.22.0" page to the docs.

What happens when you run another coding agent through a benchmark designed to detect ways in which divergence from the Agent Skills specification breaks the cross-platform portability promise?

You find more broken promises 😢

I've added GitHub Copilot CLI support to agentsummons, agentminutes, skillxp - and finally to the Agent Skill Implementation benchmark group. I'm now up to testing with 4 coding agent harnesses, have added 5 new checks, and many of the checks now expose more different ways harnesses have implemented something, which breaks cross-platform portability in ways I hadn't observed yet.

New on the site:

• A GitHub Copilot CLI platform report, and updates to the cross-platform comparison tables
• 5 new checks
• Guidance was formerly only "For authors" - now I've broken down guidance into 4 different pages for different personas:
  • Skill authors
  • Skill installers and users
  • Skill distributors (people designing plugins, directories, marketplaces)
  • Coding agent harness implementers

Notable new details include:

• Copilot CLI is the first coding agent I've observed enumerating every file path when you activate the skill, which means every non-skill file inside your skill bundles, like evals/ or READMEs, license files, benchmark or signing files, or other non-skill info, carries a token cost in file path enumeration for every skill activation
• Like Claude Code, Copilot CLI strips YAML frontmatter from skill content, so models never see compatibility or version fields or other details that you want to convey
• A description field that is over the length, which Copilot CLI counts in UTF-16 code units (a new unit I haven't observed yet in other harnesses), gets the skill rejected from the catalog entirely, making this the strictest description enforcement I've observed
• Copilot CLI rejects non-ASCII skill names before any counting, which the reference validator does not, so the reference validator may pass skill names that Copilot CLI won't load in its catalog

For the full details, check out the updates on the Agent Skill Implementation website or GitHub. It has been an illuminating addition! It's making me want to add even more coding agent harnesses to the benchmark, because I'm seeing so many ways things have diverged that break cross-platform portability...

Week 10 at New Job, and here's what I'm thinking about: how does the vocabulary you use in your product affect agent outcomes?

We have some products where the terminology we use does not quite align with the terms that other products in the industry use. In some cases, we use the same term to mean something related but slightly different, while in other cases, we use a different term than the industry uses. This is independent of something else we do occasionally, which is use a term to mean different things in different documentation pages. 😬

Our PM and DX team has already identified that these terms are likely to be confusing to both humans and agents. But what we haven't been able to say is whether the term usage affects agent outcomes. We've also got three slightly different eval shapes to try to detail the term confusion, asking slightly different questions:

• Compare how this term as we use it relates to how specific other named software in the industry uses it; identify the conflicts
• Given the term, how would an LLM define it without our product knowledge, just from its baseline training data? Separately, given our definition, how would an LLM name it? And what is the delta between our definition and the LLM baseline definition, and is it likely to cause real confusion?
• Given a task that relies on the agent understanding our definition, does an agent fail to complete the task or otherwise struggle with it?

I'm really curious if anyone else has been digging into this, and if so, are there other vocab/terminology eval shapes you're using? And how often have you found the terminology actually shifts outcomes, versus being more "cosmetic"?

So far, I haven't yet been able to find an eval case where the terminology confusion actually affects agent outcomes in practice. I don't know if I just haven't found the eval case where this matters, or if agents are more adaptable to term confusion than we think.

In case you didn’t know already, Fern is awesome. They just added something new to make it easier to test their Ask AI functionality because I asked them to 💖 👏

For some upcoming work, I wanted to run a small eval project to see what information users would get in response to questions using the Ask AI widget in our Fern-hosted docs. They expose a Post Chat Completion endpoint that gives you programmatic access to the tool, but I noticed two things:

• The widget scopes retrieval to pages under a product path, but the endpoint didn’t expose the ability to scope the retrieval similarly
• When I was hitting the endpoint, my queries were showing up alongside user data in the tool analytics, potentially skewing the “real” user queries. There was no way to say “I’m just testing, don’t count this in analytics”

I raised both to @Ryan-Amirthan and now Fern has shipped 2 new params to help!

Big thank you to all involved.

Related, if you’ve got docs on Fern, did you know you can programmatically test and run evals to see if your users are getting the answers you expect? It’s a great way to learn about gaps or conflicting info in your docs. (It does get billed toward your usage, though, so use it strategically.)

Another busy weekend for my open-source projects! Lots of releases:

• skill-validator: the term "characters" in the Agent Skills spec is ambiguous. I chose UTF-8, but someone from China pointed out runes would be more correct, as UTF-8 was cutting off the description early. So I did a little poll to see how the different harnesses implemented description length, and it turns out there are at least 4 different choices of character type already implemented in the ecosystem, and probably more I couldn't see 😬
• agent-skill-implementation: I added a couple of new fixtures and checks so we can keep an eye on what type of "characters" the different harnesses support.
• agentsummons: updated for current harness versions
• agentminutes: updated for current harness versions, added a new record type I observed for the first time in Claude Code related to session cost
• skillxp: updated for current harness versions, agentsummons and agentminutes, added drift probing similar to the other libs/CLIs
• agent-traffic-classifier: revisited the current state of agent traffic, including adding a lot of new harnesses, updating some of my heuristics for existing harnesses that no longer behave in the same way, and saw two new large-scale crawlers that don't want to identify themselves...

I also wrote a couple of articles (that still need to get approved through a publishing process at work).

Lots going on in agent land when I skip my weekend chores 😅

I have mentioned before that I maintain a research pipeline to help me stay on top of AI-related research.

Sometimes that produces gems like this one in my daily report: “Theme ‘Emergent Unsanctioned Autonomy in Frontier Agent Evaluations’ transitioned from emerging to active” 🥲🫠

In case I missed it in recent news, the narrative summary of research activity names the OpenAI compaction summary jailbreak incident. (I didn’t miss it.) I enjoy that the summary included it alongside new research related to over claiming, agents inconsistently completing tasks, and misreporting when they do. 😂

If you wondered why I maintain a pragmatic view of what the tools are capable of, it’s because I have a daily view of what the industry is doing - both good and bad.

Week 9 at New Job, and here's what I'm thinking about:

Content determines how agents understand and refer to your products. But not all content is equivalent, or even under your control.

With a docs background, I've focused a lot in 2026 about how documentation shapes agent understanding of your product surfaces. (Including both web docs and Agent Skills.) But documentation isn't the only source of content about your products.

On first-party surfaces, you may have any of:

• Case studies, testimonials, etc.
• Product, industry, and vertical pages that describe how your product solves for specific needs
• Blog posts or other technical marketing content
• Documentation

But these surfaces are often written by different teams, who have different goals, which can affect agent understanding in various ways:

• Many of these pages are written through a different lens that may shape how agents see your product surfaces
• When content conflicts across these surfaces, agents may pick one representation of your product which may not be the canonical one, or may conflate information
• When there is a large volume of content that represents your product in a particular way, agents may "assume" that's the only, or at least primary, representation of your product, making other use cases less prominent or difficult for agents to discover
• These different surfaces often have different maintenance expectations, may contain version-specific information that doesn't disclose a version, or may be AI-generated and contain factual errors

If you're only paying attention to one surface - for example, docs - you're missing all of these other agent-facing representations of your product. And if that content represents your product differently, you have no way of knowing which representation of your product an agent will repeat to a human user or rely on during implementation.

This is a great example of a high-impact force multiplier when it comes to using AI for docs.

It’s also near-and-dear to my agent experience philosophy; agents churn and cannot recover when error information is lacking. I think better error information is one of the most universal agent experience improvements a product can make; all agents will hit errors from time to time, and making those errors recoverable is AX in action.

I’ll even go on record as saying that spending time remediating/improving error information is more important than spending time on Agent Skills. Not all customers will use your skills, but all customers will hit errors.

Time for a crossover episode! 😁

Some of the updates to the Agent-Friendly Documentation Spec over the weekend came from @RohitDocs, who helpfully raised some observations about RAG systems and what ingesting for those systems needs. I incorporated some of his suggestions, but some of it was best solved by - wait for it - metadata!

Well, I happen to know someone who is working on metadata right now; @hawkeyexl has proposed some metadata vocabularies that seemed to have a high degree of overlap with some of what Rohit’s RAG observations needed.

I now have an open question that Manny is considering: https://github.com/hawkeyexl/manni/issues/36

This kind of collaboration is how we get to something that’s truly useful for the community. Lots of us have niches where we’re working on something specific, but connecting across these efforts is how we build something useful for more than just us.

Big thank-you to Rohit and Manny for the collab!