Interactive eBook BuilderInteractive eBook Builder
$59 · unlimited access, unlimited eBooksGet it for $59

· By · 9 min read

Self-Consistency Prompting: Compare Several Answers

promptsaiwritingebook

Self-consistency prompting means sampling several reasoning paths for the same question, then selecting the answer that shows up most often. In Self-Consistency Improves Chain of Thought Reasoning in Language Models, Xuezhi Wang, Jason Wei, Denny Zhou and colleagues at Google Research described this as a replacement for greedy decoding, which takes only the first, most likely chain. Their idea is that a hard problem can be solved by more than one correct path, while errors tend to scatter. The method “marginalizes out” the different paths and keeps the consistent final answer.

The paper reported large gains on reasoning benchmarks when self-consistency was added to chain-of-thought prompting, including +17.9 percentage points on GSM8K, +11.0 on SVAMP, +12.2 on AQuA, +6.4 on StrategyQA and +3.9 on ARC-challenge. Those are the authors' reported improvements on those datasets, not writing-quality scores. They are a reason to try multiple runs when a book task has a checkable answer, not a reason to vote on every paragraph.

This is also why ChatGPT can give different answers to the same prompt. The model is sampling, not retrieving a single stored reply. Self-consistency treats that variation as a signal: if most runs agree, you have more reason to trust the shared answer; if they disagree, you have a decision to inspect.

For eBook and workbook projects, Interactive eBook Builder is a $59 one-time Notion AI template with Spec Builder, Drafter, Polisher and SEO BOT already set up. A consistency check belongs in planning or in a quiz-key pass, not in every drafting run. The Prompt Engineering Lab is a natural place to run the same brief three times and compare the answers. Notion AI access and usage are separate from the purchase. The examples below are teaching prompts written for this article. They are not paid Builder prompts, and they are not recorded model outputs.

If you need a reminder of the base method, start with chain-of-thought prompting. Self-consistency sits on top of it. If the runs disagree because the brief is vague, fix the brief with a small test-and-improve loop before you take a vote.

When does a majority vote help a book, and when does it mislead?

It helps when the answers can be compared as the same kind of thing:

It misleads when the “answer” is a matter of taste or framing: which metaphor is livelier, which chapter title is punchier, or which voice sounds more like you. In those cases, a majority vote can average the text toward generic language. It also misleads when the question is underspecified. Three fluent but different outlines may all be consistent with a vague brief. The vote will pick one and hide the fact that you never defined the job.

Wang et al. describe the method as especially suited to problems with a unique correct answer and multiple ways to reach it. That is a good filter for book work.

How many times should you rerun a prompt?

The research method samples a diverse set of paths, sometimes many of them. In a chat window, three to five runs is a practical start. If all three agree, you can stop. If they split, do not immediately take 2-1 as proof. Read the disagreement. A 2-1 vote that papers over a conflict in your notes is not consistency. It is two runs making the same mistake.

Keep the prompt fixed. If you rewrite the question between runs, you are no longer doing self-consistency. You are doing a new experiment.

Example 1: check a claim against your notes, three times

A single run can sound sure and still be wrong.

I will paste a claim and the only notes you may use. Decide whether
the claim is supported, contradicted, or not mentioned.

Notes:
- The only published figure is a $90 floor for a one-page logo
  revision.
- I have no client survey.

Claim: "Most new designers undercharge by about 40 percent."

Think through the comparison, then end with one of these labels only:
Supported, Contradicted, or Not mentioned.

Run that prompt three times, or paste it into three fresh chats. If the label is “Not mentioned” every time, you can delete the claim or move it to a “need evidence” list. If the runs disagree, read the reasoning: one path may have treated a vibe in the notes as data.

Do not ask the model to “rewrite the sentence to be accurate” in the same pass. That invites a smoother, still unsupported, version of the claim.

Example 2: agree on a quiz key before you publish the quiz

Workbook quizzes are a good fit because there is a discrete answer.

Here is a multiple-choice item from my workbook, and the chapter
passage it is based on.

Passage:
[paste the short passage]

Item:
What is a floor rate in this chapter?
A. The average fee in your city
B. The lowest fee you will accept for a defined job
C. A three-tier package menu

Give your reasoning, then end with "Answer: A" or "Answer: B" or
"Answer: C." Use only the passage. If the item is ambiguous, say
"Ambiguous" and explain which wording to fix.

If two runs say B and one says Ambiguous, inspect the item. The minority run may have found a vague stem. A vote that ignores “Ambiguous” can ship a bad question.

Example 3: do not vote on voice

This is the misuse to avoid.

Write three closing paragraphs and pick the best one.

The model will pick something. That pick is not self-consistency in the research sense, because there is no unique correct close. If you want options, ask for three closes and choose one yourself, or score them against a rubric you wrote: length, no new claims, includes the fill-in line. A rubric is a judgment. A majority vote on style is not.

How is this different from a tree of thoughts vote?

Both methods compare more than one path, but they are not the same tool. Self-consistency repeats one question and looks for agreement on a final answer. Tree of thoughts prompting generates alternative next steps, scores them, and continues down a chosen branch. Use self-consistency when the answer should be unique, such as a quiz key. Use a tree when you want distinct options, such as three different outlines, and you will choose among them.

If you take a majority vote on three outlines, you may also be discarding a better minority structure. That is the wrong use of consistency. Read the options, then pick. The vote is for answers that should converge, not for plans that should differ.

How should you record the runs?

Write down the prompt, the number of runs, the final labels, and the odd one out. Spec Builder can store a decision such as “quiz item 4 key is B, three of three runs, checked against the passage.” Drafter should not be sampling five different chapter voices and averaging them. Polisher can use a consistency pass on claims and quiz keys after the prose is in place.

In the Prompt Engineering Lab, a clean experiment is: one fixed question, three runs, predict whether the answers will agree, then look at a normal item and a deliberately ambiguous item. The failure case should produce disagreement or an “ambiguous” label. If every run invents a confident key for a bad question, the brief needs a way to abstain.

If the runs keep disagreeing on a question that should be easy, the problem may be the prompt, the notes, or the item, not the number of samples. That is a recovery problem, not a vote problem.

FAQ

What is self-consistency prompting?

It is a method from Wang et al. (2022) that samples several chain-of-thought reasoning paths for the same question and selects the most common final answer. The intuition is that correct paths tend to agree, while errors scatter. It is a decoding strategy on top of chain-of-thought, not a new writing style.

Why does ChatGPT give different answers each time?

The model samples from several possible next words, so two runs of the same prompt can diverge. That is expected. Self-consistency uses the variation on purpose when the task has a checkable answer: run it several times and compare. If the task is open-ended writing, different answers may all be valid, and a vote will not tell you which one is yours.

How many times should I rerun a prompt?

Start with three runs, and go to five if the first results split. Keep the prompt unchanged. If agreement is weak, read the disagreements and fix the brief, the notes or the quiz item. More samples cannot repair a question that has no single correct answer.

Before you hit send

Consistency is evidence when the question is sharp. It is a false comfort when the brief was vague to begin with.

Ship niche eBooks with agents already built.

Get the Interactive eBook Builder for $59