Chain of Thought: Asking for Steps Before the Answer

What is it and where does it come from?

There are two basic forms. In the first, the prompt includes examples in which the answer is shown with its steps. The technique was introduced this way in 2022: Wei and colleagues, at Google, showed that with eight such examples a 540-billion-parameter model reached state-of-the-art results on GSM8K, a set of math word problems.

In the second, called zero-shot chain of thought, there are no examples: adding the phrase “Let’s think step by step” is enough. Kojima and colleagues (NeurIPS 2022) measured with text-davinci-002 that accuracy went from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K. Those are figures from a model of that generation, not a promise for today’s models.

What changed with reasoning models?

Current reasoning models build this behavior in: they reason before answering even if you do not ask (see reasoning models). The providers’ guides reflect it:

• OpenAI advises against asking its reasoning models to “think step by step”, because they reason internally, and warns that the instruction may not improve results and can even make them worse.
• Anthropic recommends general instructions such as “think thoroughly” over a hand-written plan, and keeps manual chain of thought as a fallback when internal thinking is off.
• Google documents that Gemini reasons internally, that the amount of thinking can be adjusted (thinking_level) and that only the final answer is returned by default; thought summaries are requested separately.

The practical consequence: with a model that has no built-in reasoning, asking for steps can still help; with one that reasons, it most likely just spends tokens. Prompt engineering compares the three guides; here the focus is the effect on the product.

Why does this matter to a designer?

Because intermediate steps change the interface and the waiting experience:

• Time. More steps mean more output tokens and longer waits. Show a progress state if the answer is slow.
• Trust. Visible reasoning can help verification. But it can also expose intermediate errors that confuse users who do not need them.
• Deciding what to show. You can use reasoning internally to improve the answer and show only the conclusion and the evidence behind it.
• Cost. Every step costs money. If most queries are simple, asking for reasoning on all of them raises the product’s cost with no visible benefit.

When NOT to use it?

When the task is classification, writing, direct extraction or retrieving a fact. Reasoning there adds noise, and there is evidence that it can hurt:

• Sprague and colleagues (ICLR 2025) reviewed the literature and found that chain of thought helps mainly on math and logic, with much smaller gains elsewhere.
• Liu and colleagues (2024) tested tasks where deliberating makes human performance worse. On three of six, state-of-the-art models did worse with chain of thought, by as much as 36.3 percentage points in one case.

It is also no guarantee of truth. Turpin and colleagues (NeurIPS 2023) showed that step-by-step explanations can hide what actually influenced the answer: when the prompt was biased, models changed their answers and their reasoning did not mention the bias. Later work on reasoning models found that their reasoning often does not mention the hint they used. So well-reasoned text can still reach a wrong conclusion, which is why you should verify the result and not only read the steps.

What are the ways to ask for it?

How to use it well

A practical approach:

1. Identify the multi-step tasks in your product: calculations, chained rules, comparisons.
2. Test with and without reasoning on a set of real cases, and compare accuracy, time and cost (see what are evals).
3. Ask for a structure, for example “first identify the data, then apply the rule, finally give the conclusion”. That makes the reasoning easier to verify.
4. Separate reasoning from answer in the output format, so you can show one without the other.
5. Verify the final result with code when the value is exact, such as a total or a date.
6. Test again when you change models. What helped an older model can get in the way of one with built-in reasoning.

Example

Illustrative example: an internal tool checks whether a purchase flow meets five chained discount rules. With a model that has no built-in reasoning, the team tries 40 cases and sees that in several it applies rule three before rule two. They add an instruction to evaluate the rules in order and give the conclusion last; the error drops, but the answer takes longer. The team decides to show the conclusion and, below it, the five rules evaluated with their results. A designer can review the logic without reading a long text. The exact discount is calculated in code, not by the model.

Common mistakes

• Asking for reasoning on tasks that do not need it.
• Keeping “think step by step” in prompts for a reasoning model out of habit.
• Trusting that well-written reasoning guarantees a correct answer.
• Showing intermediate steps to users who only need the result.
• Measuring only quality and not the added time and cost.
• Using the technique for values that should come from code or a database.

Sources

• Chain-of-thought prompting elicits reasoning in large language models, Wei et al. (arXiv:2201.11903)
• Large Language Models are Zero-Shot Reasoners, Kojima et al. (arXiv:2205.11916)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning, Sprague et al. (arXiv:2409.12183)
• Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse, Liu et al. (arXiv:2410.21333)
• Language Models Don’t Always Say What They Think, Turpin et al. (arXiv:2305.04388)
• Reasoning Models Don’t Always Say What They Think, Chen et al. (arXiv:2505.05410)
• OpenAI: reasoning best practices
• Anthropic: prompting best practices
• Google: Gemini thinking