Forgot to post last Friday about what I was thinking about last week, but it was eval design, from two angles.
First, I ran the first week of my eval design workshop with my pilot cohort of 10 attendees. It taught me a lot about my theories about what people should learn and what their expectations were coming in to it, and how those things differed.
I spent the weekend recalibrating the workshop around a different concept to hopefully make it easier for attendees to follow the lessons.
But the other thing I will continue to struggle with is how to balance “people want simple rules to apply” with the reality of “eval design is so conditional to what you want to know and what you can accept.” I keep saying “it depends” and trying to provide examples of why it depends, but people want simple rules to follow.
The other eval angle I spent too much time thinking about last week is people writing evals with AI. In these cases people tend to miss a lot of nuances, or make claims that the evals can’t actually support if you dig into details. So now I’m working on a checklist which will maybe become a standard to evaluate evals 🫠