Microblog

Week 5 at new job, and here’s what I’m thinking about: what does MVP look like for a team that is new to performing product evals?

Is it - a set of eval cases covering install, a handful of use cases, and one or two recover paths? Should it include “discover” and “evaluate” tasks; I.e. how much is a human deferring selection of technology/solution today, really? Do we introduce the idea of conditions or specific task completion paths; I.e. “install using pip” vs. “install from GitHub/local build” - or leave it open-ended?

And does it need to include considerations beyond user task? I was chatting with @rhyannonjoy and they brought up safety considerations. Does an MVP product eval case set need to map the idea of “safe” and “unsafe” agent actions and include evals that consider that? How much is agent safety a product consideration, versus being a harness or policy consideration?

My gut says this is all valuable, but some of it is beyond MVP for teams who are new to performing evals… but if something beyond MVP is needed, who sets up and maintains that, and when?

And of course, what does it look like to operationalize all this? And what rough edges do we need to smooth over so teams don’t have to become an expert in EVERYTHING? 🤔

Another random “agents are using your stuff in weird ways” observation: I’m running some evals, and in one run, Codex visited a single documentation page 7 times with different queries to find the information it was looking for.

Saying it again for the folks in the back: agents don’t “read” like humans do.

At the end of Week 4 of New Job, here's what I'm thinking about: agents don’t use your product like humans do, OR how you expect them to.

It’s the age old “you design a product to do X, user does Y, Z” - but for fun, agents also do “L, and something that looks like N-dimensional Foo.” It’s wild watching what agents decide to do.

For example, I’m working with a team that expected an agent to use a library by either:

• Running a provided Python file
• Running a provided tool

Instead, agents mostly:

• Wrote and executed their own Python files
• Wrote Python in a shell script

If you’re evaluating a trajectory to decide what “good” looks like, it would be easy to look only for the former and entirely miss the latter. And if you supply build scripts and tools, agents may never actually use them.

In another example, I was exploring how different interventions could improve agent accuracy and performance. I wanted to test each of these conditions in isolation, and then check some together to see what the agent would reach for when multiple options were available:

• Baseline (isolated, no augmentation available)
• Existing documentation
• Modified documentation
• A custom CLI
• A JSON file with machine-readable content

I disabled harness web fetch tools for the baseline test; one harness mostly failed, but another suspiciously got the task right every time. On inspection, it was working around the disabled web fetch tool by importing Python’s urllib and fetching documentation that way. So I thought I had designed a no-docs test, and it mostly worked for one harness, but a different agent worked around it.

If you’re designing agent experiences without testing how agents use your product in practice, you may be designing for a case that never happens. Agents use your product in weirder ways than you’d expect. And when you’re trying to isolate conditions for evals, don’t forget to check for how agents work around your constraints.

Thinking more about my Fable iOS app development project over the weekend; it was close to frictionless. But there are still things you have to do in the App Store Connect or CloudKit web UIs that agents can’t do. The handoff from “agent wrote the code” to “human has to take steps to deploy it” is now the point where friction is introduced.

That’s the next horizon for “agent experience” and I want that to be solved already.

If your product surface has actions that require a human to log into a web UI and click buttons or configure things, find a way to let agents do it. And then make sure they know how.

Fable built me a new app over the weekend. It took a few hours of iteration, mostly because I didn’t know exactly what I wanted well enough to specify it up front.

Aside from some minor CloudKit Sync issues, and some stuff you have to do in Apple Developer UIs, I did nothing except chat and everything just worked on the first try. I did read the code for quality/security stuff and it’s very reasonable.

This is a jump from the last app I wrote with AI. These things continue to get better. Unlike the last time, Fable proactively launched the simulator and inspected the UI through screenshots. It built scripts to generate sample data for UI iteration from real data I had on disk. It read existing CI pipelines to figure out how the data was generated and found a minimal, no-fuss way to fit a step into an existing pipeline to generate a version of the data that this app could use. Even just 6 months ago, I would have had to do so much of this myself.

I’ve spent this year genuinely believing this is the direction that software development is moving, but it’s wild to see it in practice.

(The app is something I won’t publish beyond private TestFlight; it compiles a reading list for me and gives me a native view of some data from a daily report I generate for myself from a series of CI pipelines that run overnight. A new CI step emits as JSON the same data that powers a web dashboard for me. Now I can read my morning briefing from my iPhone or iPad, and read state syncs between both.)

At the end of Week 3 of New Job, here's what I'm thinking about: eval design.

Engineers are no strangers to writing tests. So I think a lot of people assume engineers will automatically be able to write good evals. But I think the skills are not exactly the same. There are new things to learn.

So now we have a learning design exercise. How do we help people learn how to write good evals? What tools do they need to kick-start the process? How do we scale this to teams who haven’t already been thinking about evals?

And related, how do we get evals embedded as part of the “definition of done” in the product development cycle? Teams are busier than ever, so asking them to learn a new discipline and then embed it in their day-to-day workflows seems like a big ask. But it also seems like table stakes for any product that wants to be agent-friendly.

So this week’s agent readiness mindset is all about learning design and change management.

Your Agent Skill isn't the same on every platform. The Agent Skills spec is un-versioned and under-specified. Harnesses may have implemented support for skills before official implementation guidance landed, and may not have kept up with spec guidance. And that means that each harness may have its own story of how it supports skills, with no way to communicate to users what that looks like.

I've automated 38 of my 40 checks, with every finding cited to an archived transcript rather than the model's self-report, and I can now tell you that there are observable differences in how Antigravity CLI, Claude Code, and Codex CLI support skills, including:

• Antigravity and Codex CLIs use the skill name from the frontmatter in SKILL.md, while Claude Code uses the directory name. If the names don't match, your skill may answer to different names on different platforms, potentially breaking docs references or references in other skills.
• Claude Code strips frontmatter entirely. Aside from its name-and-description-only listing, other frontmatter fields such as compatibility notes, metadata, and any custom fields you add never reach the model through any platform channel.
• Name collisions contradict the official guide. The implementation guide calls project-over-user precedence "a universal convention". But Claude Code resolves user-over-project, and Codex doesn't resolve it at all (the model picks).

For a full breakdown of how each platform handles skills, including which ones are spec compliant and where they diverge, as well as authoring guidance based on my findings, check out the updated site.

Links from the comments:

• https://agentskillimplementation.com
• https://agentskillimplementation.com/platforms/
• https://agentskillimplementation.com/guidance/

I’ve been chatting with a few folks lately about how LLM writing is just… not very good. They use a lot of words to say very little. And we’ve all gotten to the point where certain words trigger us because we see them in AI slop everywhere.

But. I also use AI to help speed up my writing process sometimes. I scaffold content with a quick sketch of my ideas and sources to pull from, and let AI help me fill in details.

And then I edit judiciously because of those speech patterns, trigger words, and verbosity.

I finally decided to do something about it and hook up Vale in some of my personal projects, with some rules designed to target AI writing. I’ve pulled some from third party rule sets that folks have published, and made some of my own. I’ve set it up with a pre-commit hook so no new or edited file with rule violations gets committed.

Style guides in markdown, like in skills or rules in AGENTS.md, are just suggestions agents can ignore. Linters that block commits force agents to follow the rules. If you use AI for writing, lint your prose.

https://vale.sh/

At the end of Week 2 of New Job, here's what I'm thinking about: evals.

A lot of people lean heavily on evals as a tool to assess agent task completion performance. But we don't have a shared idea of what "good" looks like. People are indexing on different assessment criteria, tracking different metrics, relying on different evaluation conditions, or working with custom evaluation frameworks that may not follow design best practices. So Team A's evals may be very different than Team B's evals, and as a person trying to read eval results and take some action, those differences matter - and many of them may be invisible to the person reading a report.

Separately, in terms of thinking of evals as an agentic analog to unit tests, I'm thinking about how we can be sure the evals are running under a representative range of conditions. Just as in a unit test situation, we can't only run a given evaluation under the happy path conditions. We need to be thinking about the range of conditions under which our customers may be using our products; brownfield projects, greenfield projects, with properly-configured tools, with mis-configured tools, with only a subset of tools - and understand how agents perform in each of these conditions. A single task with a single environment configuration and no conditions isn't a representative picture of how agents perform.

And finally, I'm thinking about how we connect evals to the other facets of the DX process. If you do work with user journeys, jobs-to-be-done, personas, and other exercises mapping DX surfaces - how do we connect evals to those mappings to make sure the DX surface is well covered? Agent experience needs to work within agent capabilities and harness constraints, but it isn't completely independent from the rest of your DX life cycle, and those connections should be considered in eval design.

Random observation: FastAPI has become the default vibe-coded server. My agents basically always reach for Python/FastAPI when spinning something up that needs a server unless I specify something else, and now that I’ve seen it in a handful of my own projects, I am basically seeing it everywhere.

I would love to dig into why agents reach for this first. I’ll bet there’s a lot we could learn about agentic product selection. I’m really curious how much it is just - this is well-represented in GitHub data/public repos that a lot of agentic coding knowledge is probably based on. Or whether FastAPI inherently has any characteristics that make agents find it “easier” to use and reach for it as a default. 🤔