Microblog

Forgot to post last Friday about what I was thinking about last week, but it was eval design, from two angles.

First, I ran the first week of my eval design workshop with my pilot cohort of 10 attendees. It taught me a lot about my theories about what people should learn and what their expectations were coming in to it, and how those things differed.

I spent the weekend recalibrating the workshop around a different concept to hopefully make it easier for attendees to follow the lessons.

But the other thing I will continue to struggle with is how to balance “people want simple rules to apply” with the reality of “eval design is so conditional to what you want to know and what you can accept.” I keep saying “it depends” and trying to provide examples of why it depends, but people want simple rules to follow.

The other eval angle I spent too much time thinking about last week is people writing evals with AI. In these cases people tend to miss a lot of nuances, or make claims that the evals can’t actually support if you dig into details. So now I’m working on a checklist which will maybe become a standard to evaluate evals 🫠

Woot! After more time than I would have expected spent refining it, I've revived my not-touched-in-months TestFlight beta app, Issuenator, and added an MCP server.

Now I can do things like fire up Claude Code and say "add my public repos that I'm not already watching to Issuenator" and Claude can hunt them down and add them. Or I can ask Claude to summarize new activity, apply private local-only tags, create issues, or help me decide what to work on next based on the issues folks have filed across all of my repos.

No more weekly check of all my open source repos, or forgetting to check and leaving issues sitting for weeks (or months 😅).

Gonna live with this for a bit and then I swear this one is going to get released to sit alongside PR Focus, instead of dying in a TestFlight beta that I never actually release.

Another weekend, another big round of updating the various libs and tools I maintain:

• afdocs: v0.23.0 fixes some locale-handling bugs and adds support for script-subtag locale codes
• agentsummons: v0.4.1 updates for current harness versions, including some new flags in Antigravity CLI and GitHub Copilot CLI
• agentminutes: v0.9.0 updates for current harness versions, including changes to Claude Code API shape, support for a Codex compressed rollouts, and renamed Antigravity CLI tools + new usage information available
• skillxp: v0.4.1 updates for current harness versions and revalidates skill loading lore
• agent-traffic-classifier: v0.7.0 includes some more triggers for identifying something as a spoofed browser, which should register as bot activity, not human activity. Also more probe patterns for filtering, and more entries to the bot and scanner lists.

I've also updated Agent Skill Implementation for the current versions of the harnesses, and found for the first time since starting to run automated check in July that two of the harnesses have made harness changes that affect skill behavior. I've added a changelog to the site here: Changelog

So not only does skill behavior vary by harness, but also by harness version. Something to be aware of and make sure you're testing for when you distribute skills. (And this is independent of model behavior, which also affects skill behavior in some instances - the changelog distinguishes model behavior from platform behavior.)

This week has ranged far and wide, so it's hard to pick just one thing that I'm thinking about in agent experience land... so let's make it meta: product teams need help learning how to think about all the pieces and how they fit together to form a strategy.

Agent primitives are a big part of an agent experience strategy; everyone needs to have some idea of how MCP, Skills, and agent-related protocols may serve agents in their product surfaces. And there's also a lot to consider around how historically human-facing surfaces are now used by agents, like CLI, documentation, and even blogs and marketing materials - it's all fair game to an agent.

Product teams don't have a framework to build a shared understanding of how these primitives may apply to their product surfaces. People who are building agents are thinking about this stuff, and people who are building products that agents may use are largely left out. Ambitious or connected PMs and engineering teams have a head start in thinking about this stuff, but it’s almost entirely self-serve through various company blogs, YouTube channels, or if you’re very lucky, some research papers. But PMs or engineering teams who aren’t highly absorbed in this space don’t have good resources to bring them along. And even the ones who are thinking about this stuff may be making decisions based on incomplete information, a view of only one of these lenses, or an incorrect understanding of how any of this works.

Related, when it comes to agent experience, I am seeing some people equate one or more of these primitives as the entire strategy around agent experience. But that can’t be your entire strategy, because these primitives are all optional and for the most part, you can’t assume your customers will be using them. Some customers won’t or can’t use your MCP or Skills, and if your agent experience strategy relies on these primitives, you’re leaving out all those customers.

I don’t have a good solution for this. It’s clear that product teams need enablement materials and education aimed at where they are now so they can better understand and plan for an agent experience strategy. I am doing my best to calibrate to what teams need and provide something useful for them. But there are so many teams that need so many different things, and good enablement takes time and refinement, so the pace feels glacially slow in an area that is moving so fast and that everyone needs to understand months ago.

I'm back on my Agent Skill stuff, at least for today. Someone at work flagged a change to how a harness was handling specific skill metadata a few weeks ago, which made me realize that aside from posting about it here and on the Agent Skill Implementation website, I haven't talked to people at work about how skills that are spec-compliant may not be cross-harness portable. Or all the ways the Agent Skills spec is - under-specified - and has left harness providers to implement things their own way, which means that skills that load in one harness may not load in another.

I was heavily involved in the skills program at MongoDB; I wrote a lot of tooling, process, and enablement materials for the org. But by the time I joined NVIDIA in July, their skills program was already mature, with a dedicated PM and engineering resources, and teams shipping skills autonomously. So I've had very little to do with skills at NVIDIA.

Today I'm giving a presentation for the skills working group on these harness portability and compatibility issues. If you ship skills and you've missed my posts on this, do check out the website here where I go into details: https://agentskillimplementation.com

I'm gonna be a little bit gauche and talk about money and success for a minute.

In the last year, I've had a number of people reach out to me to pitch their project, idea, or side hustle, or to ask for business advice. I assume this is because I'm relatively prolific in public spaces - LinkedIn, my website, and in the Write the Docs community.

I think there is a shared assumption among these folks about how successful I am. So I need to tell you a few things:

• I have 141 repositories under my name on GitHub, and have been a member there since 2012. (Some of them are forks and not my own original work, but a lot of them are original.)
• I have 29 repositories under agent-ecosystem on GitHub, many of which are not public.
• I have 3 published apps on the Apple App Store, and 8 that never made it past private beta.
• I have 2 published novels (under pen names), another 2 I have ghostwritten as works for hire which other people have published under their names, and another... 8 or 9 that I never published? I've lost count.

Across all of these endeavors, I have spent tens of thousands of dollars of my own money on tools, marketing, infrastructure, and subscriptions (I would guess upward of $30-40k, ~$20k of which has been this year on all the agent-related stuff). And thousands of hours of my time working on these projects.

Between apps and book sales (mostly just apps) I have made about $15k across the last decade.

Yup, I have spent far more in costs and time to create these things than I have ever made from them. Which is totally fine, because that's not why I do it.

I do it because I want these things to exist in the world. I want to use the tools and apps I publish. I want other people to be able to use the tools and apps if they would find them helpful. And I enjoy the acts of learning, thinking, and creating. These things are some of my favorite hobbies.

But given how prolific I am, how many hours I spend on this stuff, and the fact that you've never heard of the vast majority of it... I am probably not the person to advise you about your business idea.

I think these things have all made me better at what I do for a day job, because I have been polishing skills for 15+ years that form the core of what I do for work. You could say I have developed a personal brand related to this stuff, although arguably very few people cared about that until I started talking about agents.

But I have never really tried to monetize my hobbies beyond a few cursory and sporadic marketing efforts, mostly because I know how much work, luck, and money that really takes.

If you are someone who is out there grinding and building a business, mad respect. But you're probably better off talking to other people who are grinding and building a business if you're looking for advice, because I'm out here building tools at great personal expense to myself with no plan to make money from them.

I don't recommend that as a business strategy.

Here's what I'm thinking about this week: I've said this already, but I keep coming back to it: agent experience is an extension of developer experience. (Or builder experience, however your org thinks about it.)

Yes, agents need distinct things. We know some of them, and we are still discovering others. And what agents need is a moving target as models and harnesses evolve.

But what is not a moving target is the need to understand the actual developers who are building with your product. They may be using agents to do the implementation, but the developer needs drive those sessions. If you do not have and communicate a clear understanding of who the developers are who are building with your product, what tasks they need to perform, and how your product serves those tasks, no amount of agent experience work is going to help you.

If you don't have a clear narrative for agents to understand in the context of developer goals, agents won't select your product. Period.

So all of the very hard, unglamorous product work of understanding your developers - personas and the jobs they need to do - is a prerequisite for agent experience measurement and improvement.

Yes, you can implement static checks that say whether an agent can read your documentation. But a documentation site that scores 100 in AFDocs but doesn't have what agents need to understand and use your product isn't going to get it selected. Appearing in directories and marketplaces doesn't mean agents will choose you.

DX first, then AX. Start with DX.

Nooooo! New agent rabbit hole... offline docs for air-gapped environments 😬

They don’t need the same types of artifacts that humans do, and also can’t work with web content in the same way in offline settings.

I sense a new spec in my future… 🤦🏻‍♀️

If anyone is doing anything specific re: offline docs for agents right now, I would love to hear about it.

I've had the opportunity to speak to several different folks across the industry about quality in docs in the last few weeks, and one thing keeps coming up in all the conversations: contradictions in first-party surfaces. A lot of us are wrestling with this in some form or another.

For some people, it looks like terminology challenges. Using the same term to refer to different things within or across products, or using different terms to refer to the same thing.

For other people, it looks like different claims about product capability. One document says the product can do FOO, another document omits or contradicts that claim, and suddenly you're publishing two different facts about product capability. Anecdotally, I've seen this one a lot in the past in the differences between marketing content and product docs, or point-in-time blog content not being kept up-to-date with product changes.

These types of issues don't just affect public-facing documentation and erode agent experience for your customers. They also plague internal systems and slow down your own company's product and engineering teams.

Think about your internal documents. A product team writes a proposal for a new feature in a Google Drive document. It gets implemented. Engineers document it in the code, or maybe in-repo documentation. You've got a wiki somewhere with details about the roadmap and supported features. You've got a ticketing system somewhere to track work. All of these systems expose connections for your agents, and many may now contradict each other because drift happens.

Which source of truth is canonical? And how do we communicate that to agents? And what happens if your product is versioned - how do we expose that to agents, and how are they supposed to reason about changes between versions?

If your head is exploding right now, it should be. This is one of the hardest problems around. It has been a human problem for ages, but moving at the speed of agents, these contradictions now compound so fast.

The solutions probably look different for different teams and data sources and objectives. I think we're going to be wrestling with this one for a long time to come.

Last vacation post... I have liberated my content from LinkedIn!

For ages, I've been lamenting LinkedIn's lack of an API to search my content. I've written a lot of useful stuff, but I haven't been able to have my agents look at it or do anything with it outside of LinkedIn's platform. So now I'm microblogging on my website first, and cross-posting to LinkedIn!

If you go to the front page of my website, you'll now see a "Microblog" section on the right that contains all of the relevant LinkedIn posts I've made over the last five or so years.

The microblog has a separate RSS feed, so if you want to watch just my LinkedIn-type content, or watch it in addition to my full articles, you can! And you don't have to worry about missing things that LinkedIn's algorithm just doesn't tell you about.

Every post has a cross-link to its LinkedIn version, so if you want to react or leave comments, you can click through and do so.

I've backfilled it by requesting an export of my data from LinkedIn, and I had Claude help me curate them to omit anything that's specific to events I've attended or other things that aren't really relevant outside of a time-based feed, but to keep the long-term content.

I've also added handling for things like LinkedIn's custom URL links, images, and people's full names, which might be appropriate on LinkedIn but maybe not on my personal website.

If this sounds like something you'd like to do, too, I've got an article draft scaffolded about how I did this so you could do something similar! I'll try to get it published soon-ish.