Why did the same prompt work differently today?

Watch the briefing

Watch the public video on YouTube

Full briefing transcript

The same inventory request labels a lamp electrical in one mock answer and household in another. Did the stock change? Not in our example. Both outputs use the same invented items, but their labels differ. Generative responses can vary. Rather than expecting identical wording or categories, define what consistency your task needs and test that behavior against a small set of inputs.

Our invented inventory has a desk lamp with a power cable, a wool scarf, and a ceramic mug. The initial request says only to categorize these items. One illustrative output uses electrical, clothing, and kitchen. Another uses household, accessories, and household. Neither is a recorded model run; we wrote the pair to show how an underspecified category task can accept several plausible label systems.

For an informal brainstorming list, that variation may not matter. For a stock sheet that groups purchases by fixed categories, it can matter a great deal. The useful check depends on the job. Do not confuse different wording with a factual failure, or assume a familiar label is correct because it appeared yesterday. Ask what downstream step relies on that label before deciding how strict the rule should be.

We revise the fictional request to allow only electrical, clothing, or kitchen. A powered lighting item belongs in electrical; wearable fabric belongs in clothing; a drinking vessel belongs in kitchen. If an item does not fit, return needs review instead of inventing a category. These are editorial rules for this exercise, not universal retail definitions. We want stable values, not necessarily the same surrounding sentences.

Our corrected illustrative mapping is desk lamp, electrical; wool scarf, clothing; ceramic mug, kitchen. Check a second illustrative mapping against the same rules: the lamp is household. That fails the allowed-label rule even if a person would understand the word. A representative trial with an actual tool would need its real inputs, outputs, date, model or tool details, and settings preserved. We have not performed that trial here.

When a repeated task changes behavior, first compare the source, prompt, examples, tool version, and relevant settings if you have them. Differences there may explain a changed result; unchanged inputs still do not guarantee identical output. Repeat representative cases and record whether your acceptance criteria pass. If reliability is insufficient for the consequence, narrow the task, improve review, or use a deterministic method where appropriate.

Test repeatability rather than expect identical output. In our inventory, the question is whether each item receives an allowed, rule-consistent label, not whether every sentence matches an earlier answer. Keep a record of actual tests when you run them, and retain human review for meaningful decisions. Next, we will build a compact test set containing clear, ambiguous, and incomplete cases.