Week 5 at new job, and here’s what I’m thinking about: what does MVP look like for a team that is new to performing product evals?
Is it - a set of eval cases covering install, a handful of use cases, and one or two recover paths? Should it include “discover” and “evaluate” tasks; I.e. how much is a human deferring selection of technology/solution today, really? Do we introduce the idea of conditions or specific task completion paths; I.e. “install using pip” vs. “install from GitHub/local build” - or leave it open-ended?
And does it need to include considerations beyond user task? I was chatting with @rhyannonjoy and they brought up safety considerations. Does an MVP product eval case set need to map the idea of “safe” and “unsafe” agent actions and include evals that consider that? How much is agent safety a product consideration, versus being a harness or policy consideration?
My gut says this is all valuable, but some of it is beyond MVP for teams who are new to performing evals… but if something beyond MVP is needed, who sets up and maintains that, and when?
And of course, what does it look like to operationalize all this? And what rough edges do we need to smooth over so teams don’t have to become an expert in EVERYTHING? 🤔