Self-Consistency Prompting for More Reliable AI Outputs

A single model run can be smart but unreliable. Self-consistency prompting fixes that by sampling several independent reasoning paths for the same question and choosing the answer that shows up most often. It is one of the simplest ways to trade a little extra compute for a meaningful jump in accuracy.

1. Introduction

Even with Chain-of-Thought, a single model response is one trajectory through a vast space of possible answers. With a nonzero temperature, you might get a slightly different reasoning chain each time you press send. Most of those chains will converge on the same correct answer — but a few will go off the rails. The catch is you usually do not know which response you got.

Self-consistency, introduced by Wang et al. in 2022, is a beautifully simple fix. Run the same prompt several times with sampling enabled, collect the final answers, and take a majority vote. The wisdom of crowds — applied to one model talking to itself. In the original paper, self-consistency added between 4 and 18 accuracy points on arithmetic, common-sense, and symbolic reasoning benchmarks compared to plain Chain-of-Thought.

2. The Concept Explained

The core insight is statistical. When a problem has one correct answer but multiple valid routes to that answer, sampling makes the model take different routes each time. Most routes lead to the correct answer (the "valley" in the probability landscape). A few unlucky routes lead elsewhere. If you sample five or seven completions and tally the final answers, the correct answer almost always wins the vote — even when individual responses occasionally get it wrong.

It is the same logic as asking five different colleagues to estimate a number independently and taking the median. The errors are uncorrelated; the truth shows up in the middle.

3. The Problem Without Self-Consistency

Consider a tricky word problem where the model is right roughly 70% of the time. If you take a single answer, you have a 30% chance of being wrong with no way to tell. The model will sound equally confident either way — that is the worst kind of error in any system that depends on correctness.

A factory produces widgets at 240 units/hour for the
first 4 hours, then output drops by 25% for the next
6 hours due to material shortage. How many widgets are
produced in the full 10-hour shift?

Think step by step.

4. The Solution: Sample and Vote

Run the same prompt N times with a nonzero temperature (around 0.7 is a good default), extract the final answer from each response, and pick the answer that appears most often.

prompt = """
A factory produces widgets at 240 units/hour for the
first 4 hours, then output drops by 25% for the next
6 hours. How many widgets are produced in the 10-hour
shift?

Think step by step, then write the final number on a
line that starts with "Answer:".
"""

answers = []
for i in range(7):
    response = model.generate(prompt, temperature=0.7)
    final = parse_after("Answer:", response)
    answers.append(final)

result = most_common(answers)
print(result, "votes:", Counter(answers))

5. Step-by-Step Breakdown

1. Use a CoT prompt as the base. Self-consistency only helps when there are multiple plausible reasoning paths. Always include "Think step by step" or an equivalent.
2. Set a nonzero temperature. Temperature 0 collapses every run into the same response — voting would be meaningless. Use 0.5–0.8 for genuine diversity.
3. Enforce a parseable final answer. Insist the model writes the answer in a fixed location: "Answer: <value>" or inside an XML tag like <final></final>. Otherwise voting becomes a parsing nightmare.
4. Sample N completions. Five is a useful minimum; seven to ten is better for high-stakes decisions. Beyond that, returns diminish quickly.
5. Take the mode, not the mean. For categorical or numeric answers, the most frequent value is what you want. Means can be skewed by a single outlier.
6. Surface the vote split. If the winning answer only got 3 out of 7 votes, flag it as low-confidence. Self-consistency gives you that signal for free.

Tip: Self-consistency is embarrassingly parallel. Fire all N requests at once via async / concurrent calls instead of one after the other. With most APIs, seven parallel calls finish in roughly the same wall-clock time as one.

6. Practice Exercises

7. Key Takeaways