A/B Testing Your Prompts for Better Outputs

Evaluation tells you how good one prompt is. A/B testing tells you which of two prompts is better. The technique borrows directly from web experimentation — but prompts have their own quirks, and several common A/B testing mistakes can leave you confidently shipping the worse prompt.

1. Introduction

Imagine you have two versions of a customer-support prompt. Both score well in offline evaluation, but their scores are within 2 points of each other. Which one do you ship? Offline numbers cannot answer that — they were graded on a fixed, possibly stale set of inputs. The real test is which prompt produces better outcomes on live traffic, on real users, with real downstream effects (resolution rate, escalation rate, customer satisfaction).

This tutorial covers two complementary forms of A/B testing for prompts: offline paired tests on your eval set (fast, cheap, low risk), and online traffic splits on real users (slower, more expensive, but the only way to measure real-world outcomes).

2. The Concept Explained

An A/B test for prompts has the same four ingredients as any experiment: two variants (A and B), a population, a metric, and a stopping rule.

• Variants. Same task, two prompt versions. Keep all other variables identical — same model, same parameters, same retrieval — so any difference you see is attributable to the prompt.
• Population. Offline: the full eval set, run on both prompts. Online: live traffic randomly split between A and B, typically 50/50 to start.
• Metric. A single primary metric that defines "win". Secondary metrics catch regressions you weren't tracking.
• Stopping rule. Either a sample size set in advance, or a sequential analysis with proper guardrails. Don't peek every five minutes and stop when you like the numbers — that is how teams ship phantom wins.

3. The Problem with Casual A/B Tests

Most "A/B tests" in prompt work are actually casual eyeballing: an engineer runs both prompts on five examples and picks the one they like better. That is not an A/B test — it is taste-testing with a sample size of five. Half the time the "winner" is just noise.

// Engineer:
// "Ran A and B on five tickets. B looks slightly better
//  on two of them. Going with B."

4. The Solution: Paired Offline + Real Online

# STAGE 1 — OFFLINE PAIRED TEST  (fast, cheap, low risk)
# Run BOTH prompts on the same 100-input eval set.
# Score with your judge. Look at the per-input delta.

for input in eval_set:
    out_a = model.run(promptA, input)
    out_b = model.run(promptB, input)
    score_a = judge(input, out_a)
    score_b = judge(input, out_b)
    deltas.append(score_b - score_a)

mean_delta, ci = bootstrap(deltas)
print(f"B − A: {mean_delta:+.2f}  95% CI: {ci}")

# Decision rule: only promote B to online test if
# mean_delta is positive AND the 95% CI excludes zero.

# STAGE 2 — ONLINE TRAFFIC SPLIT  (slower, real users)
# Route 50% of users to A, 50% to B. Track the primary
# product metric (resolution rate, CSAT, conversion).
# Pre-register sample size and decision rule before
# starting. Don't peek and stop early.

5. Step-by-Step Breakdown

1. Define the win metric in advance. "B wins if the primary metric is at least 2 percentage points higher with 95% confidence" — written down before the test starts. This stops you from inventing favourable interpretations after the fact.
2. Use paired tests offline. Run both prompts on the same inputs. Compare per-input deltas. Paired analysis has far higher statistical power than comparing two independent runs.
3. Pre-register the sample size. Use a power calculation, or default to a sensible minimum (online: typically 10,000+ requests per arm; offline: at least 100 paired examples).
4. Randomise correctly online. Split on a stable user/session id, not per request — otherwise the same user sees different prompts on different requests and the comparison breaks down.
5. Watch secondary metrics. Latency, token cost, refusal rate, escalation rate. A prompt that nominally "wins" but doubles cost or escalation is not actually a win.
6. Honour the stopping rule. Either reach the planned sample size, or use a proper sequential testing method that adjusts thresholds for repeated peeks. Casual peeking inflates the false-positive rate massively.
7. Roll out gradually. When B wins, ramp from 50% → 100% over a few days while watching the same metrics. Real launches surface issues that the test arm missed.

Tip: Reserve a small "holdout" segment (1–5% of traffic) that always runs the current production prompt without experimentation. It gives you a clean baseline for long-term drift detection and an instant rollback target if a winning prompt later goes sideways.

6. Practice Exercises

7. Key Takeaways