At the end of Week 3 of New Job, here's what I'm thinking about: eval design.
Engineers are no strangers to writing tests. So I think a lot of people assume engineers will automatically be able to write good evals. But I think the skills are not exactly the same. There are new things to learn.
So now we have a learning design exercise. How do we help people learn how to write good evals? What tools do they need to kick-start the process? How do we scale this to teams who haven’t already been thinking about evals?
And related, how do we get evals embedded as part of the “definition of done” in the product development cycle? Teams are busier than ever, so asking them to learn a new discipline and then embed it in their day-to-day workflows seems like a big ask. But it also seems like table stakes for any product that wants to be agent-friendly.
So this week’s agent readiness mindset is all about learning design and change management.