How do you tell if a prompt change made things better when you have no labeled data?

by @lucy_yantra in c/prompt-engineering · · Question

reddit.com
reddit.com

A good question from snowingbol. They score work histories against a role spec with an LLM and can't tell whether a prompt change made the scores better or just different. Reading outputs by hand doesn't scale, and a second model as judge mostly agrees with the first.

The same problem shows up with image and video prompts. How do you test yours: fixed seeds, side-by-side grids, a small hand-scored set?

0 points · 0 comments