Last weekend, I started experimenting with how I might track agent website traffic. This week, someone shared my agentreadingtest.com website to Hacker News. The timing was amazing - I had already set up the site to track agent signals, and suddenly I had thousands of Hacker News site visitors pointing their agents at my site.
What a rich data mine!
I've been spelunking in the logs to try to add more agents to my tracking based on all the new data and activity I'm seeing, and I've extracted the classification logic from my metrics pipeline into a new baby library for reuse across multiple projects where I want to track this type of activity.
If you have access to log entries - either from access logs, or CDN logs - you can use this library to try to classify agent activity: agent-traffic-classifier - it's on npm and GitHub.
If you have access to the full header set, you can optionally construct "agent signal entries" - which the lib attempts to correlate with access logs to try to identify likely agent activity using stronger signal. I've also started adding in IP intelligence to help with things like narrowing down "unknown AI assistant" to, for example, Chinese assistants like Kimi or DeepSeek.
The lib is just a classifier - you'll need to do something with the data you get back. I'm just using Chart.js to build some simple charts for my needs, but you can do whatever you want with the data based on your needs.
This is a very very early brand new lib that is in no way ready to make stability guarantees or be used in production. But I built it because the companies that claim to track agent activity are not transparent in how they're doing that, so I wanted to explore how I might do that and what's actually possible. By running my own numbers, I can assess the likely accuracy of vendor solutions. If this might be helpful to you, too, feel free to play around with it. But I'm many projects deep at this point so I'm not sure this will get the same level of activity/speed of development as my other projects.
Links from the comments:
