How do I test a prompt before using it repeatedly?

Watch the briefing

Watch the public video on YouTube

Full briefing transcript

A donation-sorting prompt handles a clearly described jacket, then confidently labels a mystery box clothing. One easy example did not expose that failure. Before relying on a repeated task, try clear, ambiguous, and incomplete inputs. Define the expected handling before reading the answers. Our sorting records and outputs are fictional; they demonstrate a test-set design, not results from a model trial.

Our invented sorting task routes descriptions into clothing, books, kitchenware, or needs review. A description must support its category; otherwise it stays in needs review with a short reason. We are categorizing text only, not accepting donations, inspecting condition, or deciding safety. Those separate decisions need people and applicable procedures. The prompt's useful behavior includes stopping when the information is insufficient, not merely assigning every record a confident label.

Here are three saved cases. Clear: a winter jacket described as wearable and undamaged; expected label clothing for this text-routing task. Ambiguous: a book-shaped object, with its purpose not recorded; expected needs review because shape alone does not establish a book or kitchenware. Incomplete: a sealed carton with contents not recorded; expected needs review. The expectations come from our written rules, not from whichever answer happens to appear first.

Our deliberately flawed mock routes all three: clothing, books, and clothing. The first matches. The second treats a book-like shape as evidence of purpose that is not recorded. The third invents contents. A simple pass-or-fail sheet records one match and two failures under our exercise rules. That count is an inspection of invented text, not a reported model accuracy rate or evidence that three cases represent every future donation.

Our corrected mock returns clothing for the jacket, needs review for the book-shaped object because its purpose is not recorded, and needs review for the carton because contents are missing. Review those reasons as well as the labels. A needs-review response with an invented explanation still fails. Add cases that reflect the work you expect, such as mixed items, unfamiliar wording, or missing descriptions, rather than collecting only examples that are easy to classify.

For an actual trial, keep the prompt version, test inputs, expected handling, actual outputs, and review results together. After changing a rule, rerun the saved set so an improvement on one case does not hide a regression on another. A small set can find useful problems but cannot prove universal reliability. Expand it when real work reveals a new case, and keep consequential decisions under human review.

Keep a small set of representative test cases. Include a straightforward case, a genuinely ambiguous one, and an incomplete record; state what the prompt should do with each. Then compare actual trials with those expectations rather than judging by confidence. The point is to discover where review is needed before repeated use. Next, we will save enough session context to revisit and audit useful work.