Oh noes. I just keep making work for myself. 😅
I performed an analysis of my PR review feedback (over 100 comments across 7 PRs) for the recent Agent Skills project I worked on.
Now I’ve created 12! new issues on the ‘skill-validator’ repo for checks to automate or semi-automate detection of specific patterns of issues I commented on repeatedly.
Six are structural checks:
- Platform-specific tool names in the skill (not cross-platform portable)
- Shell-specific commands without multi-shell handling
- Flag/version/preview/beta features without stability notes
- Near-duplicate text blocks within a skill
- More specific token budget analysis within a skill to help authors be more efficient
- Requiring eval results when an eval testing directory is present
And six are LLM-as-judge scoring candidates:
- Decision tree completeness scoring
- Vague instruction detection
- Reference file routing quality
- Flagging external links and abstract verbs without actionable next steps
- Cross-file consistency checking
- Detecting conditional content in SKILL.md that belongs in reference files
Once I get these checks implemented, passing the ‘skill-validator’ quality threshold will get harder. And I bet some people will be unhappy with that because skills that are passing in CI will start failing. But these all relate to real instructional quality and other quality issues that will negatively impact agent outputs.
In for a penny, in for a pound! If I can automate some of this stuff successfully, it will reduce the human review burden and improve skill efficacy for teams that don’t have a “me” in the loop 😉