Microblog

I've been getting a lot of messages lately from people asking me to post somewhere about their cool thing they have built on top of AFDocs. Rather than handling all of those as one-off requests, I've added a "Powered by AFDocs" page to the site!

If you have created a thing that uses AFDocs, and it's publicly available and attributes AFDocs, feel free to PR to add your thing to the page.

It's pretty neat to see a roundup of all the cool things people are building 🎉

This week's "what I'm thinking about this week" isn't about work, since I had the week off work! (Well, at least not directly.)

I did get some downtime in, but if you know me, you also know that when I'm not working I'm working on personal projects. So much to do, so little time. So this week I've been focused on a new pipeline to surface agent experience-related topics from across the industry.

This uses my new topicsleuth library to analyze content I feed it from arXiv, product releases, industry bloggers and reporters, and other sources I care about. It uses domain-specific rubrics to identify agent experience-related content and assess how the content relates to AX topics I'm tracking, or propose new ones.

Then, create a daily briefing that surfaces relevant content for me from high-value topics. And a weekly briefing that helps me track changes over time, and make some forward-looking projections about where the industry is going. Because this is a single-user project, I can be very specific - the screenshot below is one of the topics from today's run, with recommendations that the model made for me based on what it thinks I care about. (It's pretty good!)

It can help me track changes by vendor, can help me spot consolidation or bifurcation in the industry, and spot emerging trends or identify topics that are fading in importance. And because I've been tracking related content since I started my research pipeline in March, I was able to backfill this new project with thousands of pieces of content so I can already see changes over time.

🤞 this helps me keep an eye on emerging and maturing trends in agent experience as this new field is being defined. (And yes, it's wild and amusing to see my own content feeding these topics, too!)

My new lib is out! This one contains the logic that runs my research pipeline. To use it, you find content you're interested in and feed each piece of content through the lib. The lib:

• Uses vector embeddings to pull semantically-related topics from a vector store
• Passes the similar topics and the piece of content to an LLM and asks it to say whether the content is related to any similar topics, or warrants a new topic
• For each connection, it asks whether the content supports a topic, extends a topic, contradicts a topic, or marks a shift in the topic (i.e. something that was previously only research is now a feature shipped in a product)
• You can optionally use a second model - either an LLM or a decision model like Jev - to provide a second opinion if a new topic is being proposed
• It also tracks topic lifecycles, noting whether a topic is emerging, active, mature, fading, or dormant

It comes with the machinery to tune results based on how well the LLM is identifying content relative to your topics. I have found that it worked better with an earlier version of Opus, but a newer version was more likely to propose new topics (new topic creation went from 20% to ~40+%). So best results may require making some tweaks until the topic creation and assignment matches your human judgment, and you may need to tune things as parts of your pipeline change.

This has been working well for me in a pipeline for ~7 months, but I've broken it out into a lib because I want to use it in a couple of new ways/projects. If you have any need to watch how topics emerge, grow, and change over time, you may find it useful - look for topicsleuth on PyPI or I'll throw a link to the GitHub repo in the comments.

Week 12 at new job, and here's what I'm thinking about: how can we leverage work other people are already doing to accelerate our understanding of agent experience?

Attending the internal agent summit this week gave me a lot of great glimpses into what other teams are working on. Some of it is directly relevant to problems I'm watching product teams struggle with, such as how do you set up a realistic environment to run evals, or how do you create a realistic task set? Other people are working on helping partners and enterprises answer these questions, so how can I leverage that work for our internal product teams?

I don't know the answer yet, but it's something I need to make time to dig into.

This is also somewhat related to how I keep up with what people across the industry are doing. I've been running an AI news gathering and filtering pipeline for my personal use since March. Last weekend, I started redesigning this to bring in more sources, drop some of the work I had envisioned that never materialized, and add lenses to help me focus on agent experience. This is going to involve extracting some of my topic analysis and tracking code to a shared library, which I'm hoping to release in the next few weeks.

As the industry keeps moving, I need to keep iterating on both how I can track that progress and how I can bring what applies into both my work and personal projects.

Also, time flies! I've been here nearly 90 days so I think I'll stop the "week number" and "new job" references but keep posting weekly what I'm thinking about. It has been useful for me in spurring conversations with folks, and I hope it's helping other people think about how we can shape a shared understanding of what agents need and how we can help them discover and use products more effectively.

Protip: if I am in a meeting with you, and you’re showing off your very cool agent thing, I am going to ask how you’re testing it.

Just plan for this and be ready to talk to me about it.

If you’re testing it in a meaningful way, I will be so impressed and will want to learn from your techniques.

Please impress me… please…

I got a chance to give a lightning talk today at an internal company summit around agent things. A few things stand out from both my session and other sessions today:

• Holy heck is it cool to be working at NVIDIA at this moment in time. Lots of people are working on lots of very cool agent things. It’s really inspiring to see what other people are working on, and get ideas and potential collab for the things I’m working on. 🤩
• Agents are super cool but there are also a lot of hard engineering problems around all this stuff. It really is a generational shift in software engineering, and figuring out new ways to apply familiar patterns in this new paradigm. And availability of tokens does not solve the hard problems. That takes work.
• Even in this room of very smart people working on very cool things, the thing I am working on remains a hard problem. Nobody else has a magic fix that makes my hard problem easy. That is validating in that I think my work is worthwhile and justified - I’m not wasting effort on something someone else has an easy fix for. But also a little sad because an easy fix would let me say “ok, THIS is solved, let me go work on the other 999,999 problems I find interesting.” 🤣

Awesome day. Very curious what else I’ll learn about during day two tomorrow 😁

The number of AI-written research papers on arXiv these days really makes me sad 😢

The quality of the content varies wildly; some people edit judiciously so it’s more the phrasing that makes me cringe, but others just accept the AI spew whole cloth which means I have to wade through mountains and mountains of repetitive and duplicative content to pull out the actual points.

But more critically, after messing around for months with an AI writing pipeline, I never did manage to get it to a state where factual correctness was satisfactory. Knowing how much effort I put into that, and how many factual issues were present even after every strategy I employed to mitigate it, I have a hard time taking any of these AI-written papers seriously. If the authors can’t take the time to edit their papers I have no reason to think they took the time fact check the volume of assertions these papers make, or the underlying data they probably also generated with AI 😢

Week 11 at New Job, and here's what Im thinking about: how do I scale myself and share learnings effectively across a company with hundreds of products and 40k+ employees?

I can't be everywhere all at once. I have a multi-year research backlog if I'm the only one doing it. It would probably take me 10 years to work with every product team individually to help them learn about agent experience. Since I can't clone myself, I've gotta figure out how to scale. Here's what I'm thinking so far:

• I'm putting together a 50-500 level course to help my colleagues go from 0 eval experience to performing ablations and other types of studies
• I'm preparing a newsletter strategy to share agent experience learnings from experiments we run
• I've been running a weekly forum for people to come together and share their learnings, which I am recording and planning to turn into reusable patterns and best practices
• I'm speaking at a summit next week to raise awareness of the specific eval shape we've seen product teams struggling with, partially to let people know they can be doing this and partially to solicit feedback on how we might help product teams upskill around this
• I'm drafting self-serve enablement materials which will live in our internal information-sharing spots, and may eventually turn into tooling or other artifacts to help teams upskill on their own time, without us having to be in the loop for teams that can't wait until we have capacity
• I'm working with my teammates to draft a landing page for this work with more details so people can learn about it without needing to have a meeting with us
• I've written down a running doc of research ideas, why they matter, and what the research methodology might look like, so people who are interested in any specific questions can pick up a task and run with it

What have I missed? For other folks who have run into this problem, I would welcome any tips you have!

The more I dig into how agents fail on tasks, the more I find about how product surfaces need to change to reflect this new consumer. Let me tell you about two recent examples:

First, version information in docs. I was able to reproduce agent failures consistently related to missing version information, on multiple axes. When humans go to a versioned documentation page, they typically see version information on the page. Maybe it's interpolated into the page title or header, and usually there is a version switcher that lets humans select a different version of the docs.

Agents don't see that. When agents hit the markdown versions of the page, like Claude Code requests markdown by default through content negotiation, if the markdown content doesn't include the version information, Claude doesn't see it. And most sites I've seen are not generating version information into the markdown. Here's what this failure looks like in practice:

• Claude sees an install page that shows how to pin a version, which includes v0.5.1. Claude installs v0.5.1. But the product has moved on to v0.6.0. Except Claude doesn't see anything about v0.6.0 on the page.
• Claude looks up documentation for specific functionality. That documentation reflects functionality in v0.6.0, but it doesn't say so on the page. Claude tries to use it, but the functionality doesn't exist in the version of the product it's using, which is behind.
• This is where I've seen it diverge: Claude either tries to build its own version of the feature as written without mentioning anything to the user, or tells the user that the functionality is absent and it can't do the requested task, or implements something completely different which it may or may not call out to the user. I have never yet seen the agent recognize that the version is behind, or even check GitHub to figure out the discrepancy, even when the product is public.

So missing version information bites agents in multiple ways: it can cause the agent to install old or outdated products even when looking at up-to-date documentation, or it can cause the agent to reach for functionality that doesn't match the installed version of the software.

A second example is related to patterns that exist in training data, and how agents can confidently invent things that don't exist. I was working with a product that has a start and a status command. Claude retrieved documentation for the product, which uses its web fetch tool with a smaller model, Haiku, summarizing page content for the orchestrator model - in my case, Opus 5.5. Haiku tells Opus that the product has a start, status, and stop command.

The product doesn't have a stop command. It uses ctrl + c to stop. But Haiku is a small model, and there is enough start, status, stop patterns in its training data that it makes up the plausible command that doesn't exist. So my open question is - should the product add a stop command? 🤔

Can I just say how cool it is to:

• Ask an agent to do a task
• Hand it a link to a relevant docs page
• See it observe the in-page directive pointing to llms.txt
• Watch it pull the file, and then proceed to pull like 4 other relevant docs pages to answer questions it has

It feels so awesome to watch my tools work better/more effectively and know I played a part in shaping that outcome 🎉