Microblog

The Agent-Friendly Documentation Spec got a version bump over the weekend, bringing it to 28 checks (5 new) at v0.6 😅

The new checks include:

• bot-protection-interference: bot-protection systems can interfere with coding agent documentation fetching, so this check identifies three different forms of bot protection that can block coding agents from viewing your docs
• page-size-transfer: the spec already had page-size-html but that was a post-conversion metric; the new check is about how much of the content is serialized framework payloads, which is another form of "request is too large so coding agents can't see your content"
• single-fetch-completeness: looks for pagination in markdown responses, and verifies the continuation is machine-followable
• markdown-link-portability: links served in markdown should be absolute, not relative, and should resolve to the representation they promise - not a soft-404
• embedded-data-serialization: machine-generated static representations of dynamic content can push otherwise prose-light pages beyond coding agent truncation limits, so this check is about the size of those bulk elements

I've also added two new interaction effects:

• Bot Protection Degrading Scan Reliability: bot protection enforcement can engage mid-scan, so this interaction effect is about identifying what bots can see and measuring it before bot protection engages
• Dynamic Content Rendered Statically: this involves three of the new checks, plus markdown-content-parity and the Category 3 size checks, and is about the different ways that flattening dynamic content for agents can fail

Notably, these additions also pushed the Agent-Friendly Documentation Spec website beyond the truncation limit for agent fetches, so I've now broken the website down into separate pages so agents can consume it. Since this breaks the old URL path, I've also moved it to a web URL, officially recognizing that this is the "Web Documentation Delivery Spec" - and committing to at least two more companion specifications I've been dancing around for months:

• Content composition: what documentation should contain to serve agents well: factual consistency across pages, structure and density suited to machine consumption
• Repository-local documentation: docs serving coding agents that work inside a repository, where discovery happens through grep and file listings rather than HTTP

If you have any thoughts about what needs to go in the new content composition or repository-local documentation specs, please file issues on the GitHub repo at agent-ecosystem/agent-docs-spec - I am happy to discuss!

The afdocs CLI is not yet updated to reflect spec v0.6; I'm hoping to get those changes into the CLI/lib in the next week or two.

Week 8 at New Job, and here’s what I’m thinking about: how do you help agents understand changes to product positioning?

We’ve got products that have had substantial development in the last 6 months, but agents remain stuck at a moment in time before that development. They give users wrong information about capabilities, or understand our product incorrectly.

We’ve got some products that are intentionally changing positioning to better align with org strategy and lessons learned from other products. How do we help agents understand shifts to intended personas and use cases? How long does it take, for example, for updated docs to feed into AI model training crawlers and answer engines, and become reflected in coding agent sessions?

What do you do when some sources of information are under your control and can be updated, but other sources of information from media or third-parties aren’t under your control and reflect outdated positioning? And how does that affect agent understanding of product positioning?

I’m starting to think about how to design one or more studies to better understand this. If you have thoughts or are doing anything around this, I’m all ears!

I really want to do really cool, novel, exploratory agent experience studies and experiments. And I have faith that I’ll get there eventually.

But so far, I just keep finding myself pulled back into docs-related things.

So, PSA: before you can do all the cool stuff, you need good, up-to-date docs that agents can access. Docs are the foundations that AI model training crawlers and answer engines use to learn about your products. Coding agents refer to them in user tasks. If agents can’t access the info, or if it’s outdated or contains contradictions, you’ll have problems and nothing novel can save you.

There is a slightly weird thing about working at an AI company when you... take a pragmatic view of what the tools are capable of.

There was an exchange last week that has stuck in my head, which sorta encapsulates the whole thing.

Me: "And even when an agent can write the code, and it compiles, it is often less performant than our libraries." (Speaking of our kernel libraries.) Them: "Then tell them to make the code more performant. I've gotta believe it's capable of writing code better than a human. Otherwise, what are we doing here?" Me: "I haven't seen evidence of that."

Shortly after that, the person left the room. I don't know if it was because of this exchange, or they just had somewhere else to be.

But here's the thing. Yes, AI is great. Yes, it continues to improve. Yes, I use it all the time.

But I believe that currently, someone with domain expertise can often still write better code (or prose, or docs) than AI. Maybe not faster, but more correct, or more performant, or more aware of edge cases, or better fit for an overall system or pipeline, or any of the many things humans can still outperform AI in.

It's a great democratizer. It lets people who lack domain expertise create things they wouldn't be able to create otherwise (myself included). But I keep being confronted with cases where AI output is less than human.

Which I guess means I will keep having encounters like this with my peers.

Week 7 at New Job and this was an onsite for my team and some of the teams we work with closely.

While I love my remote life, it was great to meet people in person, see the pretty epic office, and collab on some chonky problems.

When you’re part of a team that works cross-functionally, it’s really hard to beat face time with partner teams.

We had some really interesting discussions around who is a developer now, whether developers and agents are independent users with their own journeys, and how we can better understand the impact of changes we may be making to serve one agent audience (AEO, model training crawlers) on other agent audiences (coding agents).

Also, metadata.

Great week!

Week 6 at New Job, and here’s what I’m thinking about in agent experience land:

Enabling agents to discover the right information at the right time is still a huge unsolved problem. Skills are an attempt to do that, but they have their own problems, from harness/model combinations just NOT choosing to invoke them to authors writing them at the wrong level of abstraction or filling them with unneeded/unhelpful information.

Progressive disclosure is the principle at play, with the recent “The New MCP Roadmap” announcing a progressive discovery effort for MCP tools, a problem I’ve seen firsthand with enterprise customers. Skills also attempt to provide progressive disclosure. It’s the right principle IF you can get agents to decide they need more information and actually request it.

A problem I’ve continued to see across all “agents must request more information to successfully complete a task” - whether that be load a tool, visit a docs page, or use a skill - is that plenty of times, the agent just chooses not to request more information. It proceeds with what it has, guesses, and fails, and has to spend time/tokens recovering or just fails to recover.

I’ve also been thinking about Opus, specifically; I think some of its “more expensive, slower, over-engineering” tendencies may be related to trying to solve this. I.e. I think Opus is trying to get the information it needs to complete the task successfully, but it’s manifesting as these behaviors.

So, hard problem remains hard.

Not gonna lie, I didn’t make a lot of headway with my local inference experiments with my beefy gaming PC. A beefy gaming PC with one GPU, even the best one I could buy, is not an inference workstation.

The models I could run ran lightning fast, but they were no match for the frontiers in coding and that’s most of what I use AI for.

Really curious (and tempted) to go down this rabbit hole with the new Apple hardware. Wanna see some data from people actually using it.

FYI, one run is not enough to “prove” agents can or can’t do something.

Two isn’t, either.

Not even three.

It takes a lot of runs to produce a meaningful finding.

For anyone who uses agent-ecosystem tools, I've had a busy weekend for maintenance and everything has gotten updates:

• afdocs: couple of small bug fixes and --canonical-origin now accepts a prefix path to support previews shipping from a different base
• skill-validator: couple of small bug fixes in YAML parsing and llm-judge instructions
• agentsummons: updated for current Antigravity, Claude Code, and Codex harness versions
• agentminutes: updated for current Antigravity, Claude Code, and Codex harness versions + added prompt token normalization (to some degree, the accounting is weird) across Claude Code + Codex. Notably, Codex's log format changed again so there are some changes to how you would promote certain event types.
• skillxp: updated agentsummons and agentminutes dependencies for current harness versions

Thanks to everyone who PR'd and filed issues on the tools. Really happy folks are finding them useful.

Interesting research paper dropped on the 20th about how coding agents discover, read, and write technical documentation: https://arxiv.org/pdf/2608.20195

The main finding is that agent instruction and plan files are the primary agent documentation interaction interface, with agents rarely consulting classical genres that documentation teams focus on (10.6%) and far more rarely consulting API documentation (1.3%)

BUT there are some problems with the methodology and data set. They've scoped specifically to "repository-local, file-based documentation interactions." It doesn't observe docs read through a browser, and explicit documentation-retrieval tool calls were almost entirely missing from their data set (only 16 events).

Also, the data source skews heavily toward one usage pattern. It's 83.8% Claude Code sessions, and Codex has pretty different reading characteristics. It is also entirely composed of developers who opted into one service's checkpoint logging feature on public repositories, with short routine sessions over-represented and most failed sessions never captured. This dataset self-selects into a specific usage pattern or behavior that isn't representative. They also encountered firsthand the issues that prompted me to write agentminutes to normalize session logs from different harnesses into comparable event types.

People who only read the opening numbers may conclude "agents don't use API docs" (based on the 1.3% number) but really, what they've shown is that "when working inside a repo, agents rarely read local API reference files."

There's a lot of missing context in the methodology and data set of most research papers, and it's important to dig in so you can understand what the findings really show.