Oh noes.
I have set up an evaluation system to score LLM outputs for code gen tasks involving our products, which I’ve planned to use to validate any Agent Skill content we create.
For my first test case, the Skill content made the output worse than the unassisted LLM performance. 😬 🙃
We support 12 programming language drivers, plus a shell tool and some other interfaces.
Turns out, when you shove a code example in front of an LLM, it really wants to mimic that code example, even if it’s for a different programming language. So agents that can write the code perfectly fine from base training data actually mess it up when they see a “wrong” example in context.
For products that support a lot of languages… I’m wondering if this will require language-specific Skills? 😰 😱
Also, the code example patterns that we show in docs, which are written for humans to learn from - not intended to represent production code… yeah, maybe those are the wrong examples for LLMs. 😕
This is going to be an interesting learning curve…