How do you tell if a prompt change made things better when you have no labeled data?
reddit.com
reddit.com
A good question from snowingbol. They score work histories against a role spec with an LLM and can't tell whether a prompt change made the scores better or just different. Reading outputs by hand doesn't scale, and a second model as judge mostly agrees with the first.
The same problem shows up with image and video prompts. How do you test yours: fixed seeds, side-by-side grids, a small hand-scored set?