- Chain of thought prompting was a real result on models that did not reason by default. The 2022 paper reported state of the art accuracy on the GSM8K word problem benchmark from eight worked examples.
- On models trained to reason before answering, asking for steps buys little. A 2025 study found only marginal gains and more variability on questions the model already got right.
- OpenAI's own documentation tells you to skip it on reasoning models, in one sentence, and to keep the prompt short instead.
- Every reasoning technique is paid for in output tokens. Thinking tokens are billed as output, and on Claude the minimum manual budget is 1,024 tokens per request.
- Self consistency, running the same question several times and taking the most common answer, added 17.9 percent on GSM8K but multiplies the bill by the number of runs.
- For shop arithmetic the deciding question is not which technique to use, it is whether the task has one checkable right answer. If it does not, none of this helps.
You ask the model to work out a price. Cost is 8.40, you want 62 percent margin, the platform takes 4.9 percent plus 30 cents, and the country adds 20 percent VAT on the sell price. It hands you a number with total confidence. The number is wrong by about a euro, which is most of the margin you were protecting.
The advice you have probably heard is to tell it to think step by step. That advice was correct, it is now half correct, and knowing which half applies to the model in front of you is worth more than any prompt template.
What is chain of thought prompting, in one paragraph?
It is showing the model worked examples so it produces its own working before it produces an answer. The 2022 paper by Wei and colleagues demonstrated it with eight worked examples in the prompt, and reported that a 540 billion parameter model reached state of the art accuracy on GSM8K, a benchmark of grade school maths word problems, beating a fine tuned model with a separate verifier attached. The effect only appeared in large models. Small ones produced working that looked like reasoning and arrived at nonsense.
The everyday version, the one people actually use, is the four words appended to a question. That version is a shortcut to the same behaviour and it inherits the same limits.
Why has the advice changed?
Because the models changed underneath it. A model that has been trained to reason before it answers is already doing internally what the prompt used to ask for out loud. Telling it to think step by step is asking a person who is already writing to please pick up a pen.
The measurement comes from a June 2025 report by Meincke, Mollick, Mollick and Shapiro, written for business readers rather than researchers. Their finding splits cleanly. On models without built in reasoning, chain of thought still helps, by a small margin. On models with it, gains are marginal at best, and the technique introduces extra variability that can turn a question the model would have answered correctly into one it gets wrong. You are paying more tokens for a wider spread of outcomes.
The vendor documentation agrees, which is unusual. OpenAI's reasoning best practices page states it plainly: because these models perform reasoning internally, prompting them to think step by step or to explain their reasoning is unnecessary. It goes on to recommend short direct prompts and to try zero shot before reaching for examples, which is the reverse of the advice that circulated for three years.
What does asking for steps actually cost?
Output tokens, and they are the expensive kind. On the Claude API, the extended thinking documentation sets a floor of 1,024 thinking tokens per request and requires the budget to sit below the turn's max_tokens ceiling, because thinking counts against the same allowance as the answer. The docs suggest starting near that 1,024 minimum for simple tasks and at 16,000 or more for complex ones, with returns that diminish as the budget grows.
Two details from that page are worth stealing regardless of which vendor you use. The first is that you can measure the reasoning portion of your bill directly rather than guessing at it, through a field in the response that reports how many billed output tokens were internal reasoning. The second is that changing the thinking budget between requests invalidates prompt caching, so a shop that fiddles with the setting per request pays twice: once for the reasoning, once for the cache it kept throwing away.
The general rule underneath all of this: reasoning is not a mode you switch on, it is tokens you buy. Any technique that produces more intermediate text costs more and takes longer. We worked through how those token charges assemble into a monthly figure in the piece on what tokens actually cost when you run something real.
A worked cost example, in tokens
Abstract advice about token cost never changes anyone's behaviour, so here is the arithmetic on a job a shop really runs. You have 2,000 products and you want a description for each. On a model with manual thinking set to the documented 1,024 token minimum, the reasoning alone adds 2,048,000 output tokens across the batch, before a single word of the descriptions themselves. Raise the budget to the 16,000 tokens the same page recommends for complex work and the reasoning portion becomes 32 million output tokens for a task that is not complex.
Now apply the filter from earlier. Writing a description is a judgement call with no single right answer, so every one of those reasoning tokens bought variance rather than accuracy. The same batch with thinking off, or on a fast model with a good brief, produces work you will edit either way.
Reverse the example and the conclusion reverses with it. Recomputing landed cost across the same 2,000 products is arithmetic with one right answer per row, the errors are expensive and invisible, and 2 million extra output tokens to avoid a systematic rounding mistake is a bargain. Same technique, same spend, opposite verdict, decided entirely by whether the task has a correct answer.
The documentation adds one operational warning to this. Thinking budgets above 32,000 tokens should go through batch processing, because long running requests start hitting timeouts and connection limits. Anyone planning a large catalogue job with deep reasoning turned on is planning a queue whether they know it or not.
The four techniques, and where each one belongs
The table below sets the common reasoning techniques against the task each one suits, what it adds to the bill, and the situation where it makes results worse rather than better. Assembled from the papers and vendor documentation linked throughout this article.
| Technique | Suits | Added token cost | Where it backfires |
|---|---|---|---|
| Chain of thought, four words appended | Multi step arithmetic on a model that answers in one pass | Roughly the length of the working, often several times the answer | On a reasoning model, where it adds variance without accuracy |
| Chain of thought with worked examples | A calculation you repeat in a fixed shape, such as a landed cost | The examples are re-sent on every request unless cached | When your examples encode a mistake, which then repeats forever |
| Self consistency, several runs and a majority answer | One high value number you cannot check by hand | Multiplies the whole request by the number of samples | On open ended questions with no single correct answer to converge on |
| Decomposition into separate calls | A task with distinct stages, such as extract then compute then format | Each stage is a full request with its own input | When stages share context you then have to re-send every time |
| A reasoning model with no technique at all | Ambiguous multi step problems where accuracy matters most | Thinking tokens, billed as output, from 1,024 upward | On simple lookups, where latency and cost rise for nothing |
Self consistency is the technique nobody costs properly. The 2022 paper reported gains of 17.9 percent on GSM8K, 11.0 percent on SVAMP, 12.2 percent on AQuA and 6.4 percent on StrategyQA. Those are large. They are also bought by sampling the same question many times and taking the majority answer, so a five sample run costs five times a single one. For a margin calculation you run twice a year, that is nothing. For anything on a page a customer loads, it is the whole budget.
Does any of this help with the work a shop actually does?
Sometimes, and the filter is sharper than the technique. Ask one question first: does this task have a single right answer that I could check if I had an hour?
If yes, reasoning techniques are on the table, and so is the discipline of checking. Landed cost per unit, break even quantity on a discount, a reorder point from lead time and average weekly sales, the VAT inclusive price that yields a target margin: these have right answers. A model with worked examples in the prompt will get them right more often than one without, and you can verify the output against a spreadsheet once and then trust the shape.
If no, none of this applies. A model asked to reason step by step about which product name is more appealing will produce three paragraphs of confident justification for an answer it picked on vibes. The working is generated after the fact and it is not evidence of anything. This is the most common way small businesses waste tokens: buying reasoning for a judgement call that has no ground truth to reason towards.
Which model should the task go to at all?
This question is upstream of every prompting decision and it gets asked last. The OpenAI guidance is unusually concrete about it. Send work to a reasoning model when accuracy and reliability dominate, when the problem is multi step and ambiguous, when it involves planning, or when it sits in a domain like finance or law where being wrong is costly. Send it to a standard model when speed and cost dominate and the task is well defined.
Most real workflows use both, with the reasoning model deciding what to do and the fast model doing it. For a shop that maps onto something recognisable: one careful pass to work out the pricing rules for a new range, then two thousand cheap passes to apply them. Paying reasoning rates on the second half is the most common way a small budget disappears without anyone noticing.
There is a newer wrinkle worth knowing. On models with adaptive thinking, the decision about whether to reason at all has moved inside the model, which decides per request and can skip thinking entirely on easy inputs at lower effort settings. That is good for a bill and awkward for a benchmark, because two identical prompts can now take different paths. If you are comparing outputs across a batch and the variance surprises you, this is a likelier explanation than anything in your prompt.
What should replace step by step on a modern model?
Constraints and success criteria, in ordinary language. Instead of asking for reasoning, state what the answer must satisfy. Give the model the tax rule that applies, the fee structure with its fixed and percentage parts, the rounding convention, and what the output should look like. On a reasoning model this is the whole job, and the documentation says as much when it recommends being specific about the end goal rather than about the method.
A worked contrast, using the pricing problem from the top of this article. The weak prompt asks for a sell price at 62 percent margin and adds the four magic words. The strong one supplies the unit cost, states that the platform fee is 4.9 percent of the gross plus a fixed 30 cents, states that VAT at 20 percent applies to the customer facing price, defines margin as the share of the net that remains after cost and fees, and asks for the net price, the gross price and the resulting margin as three separate numbers. The second prompt does not mention reasoning at all. It wins because it removes the ambiguity the model would otherwise have to guess at, and guessing is what produced the wrong euro.
The other substitution worth making is structural rather than verbal. Ask for the answer in a fixed shape you can validate automatically, so that a wrong number fails loudly instead of arriving in a paragraph. Our piece on getting reliable structured output out of a model covers the mechanics, and it pairs well with anything numeric.
When is a worked example still worth the tokens?
When the shape of the task is unusual and the shape of the answer matters. A model knows what a margin calculation looks like. It does not know that your business treats a shipping subsidy as a cost line rather than a discount, or that your reorder point convention counts weekends. One example carrying that convention teaches more than three paragraphs of instruction, and it survives model upgrades better than a prompt full of rules.
The failure mode is worth stating because it is silent. An example that contains an error teaches the error. If your worked example rounds at the wrong step, every answer that follows rounds at the wrong step, confidently, forever. Check the example harder than you check the output. We went further into when examples earn their place in the piece on how many examples a prompt actually needs.
A short decision procedure
- Establish whether the task has one checkable right answer. If it does not, stop, and write a better brief instead of a longer prompt.
- Establish whether your model reasons by default. If it does, say nothing about steps and spend the words on constraints.
- If your model does not reason by default and the task is arithmetic, add one worked example in your own conventions rather than the four magic words.
- If the number is expensive to get wrong and cheap to compute twice, run it more than once and compare. That is self consistency without the paper.
- Measure what the reasoning is costing you, using the vendor's own reporting field rather than an estimate.
What the research does not say
It does not say chain of thought is dead. The 2022 result stands, and there are still plenty of models in production, particularly cheaper and open weight ones, that answer in a single pass and benefit from being asked to work. It does not say reasoning models are always better either. The same OpenAI page that tells you to drop step by step prompting also says to use a standard model when speed and cost matter and the task is well defined, and notes that most real workflows use both: reasoning for planning, a faster model for execution.
What has actually changed is where the technique lives. It moved from the prompt into the model, and its cost moved from something you could not see into a line on your bill. That is a better arrangement, but only if you look at the line. A small shop running a handful of calculations a month will never notice. A shop generating descriptions for two thousand products with reasoning switched on will notice at the end of the month, and by then the tokens are spent.
If you are choosing which model to point at which job in the first place, the trade between a fast one and a thorough one is the real decision, and it is laid out with current figures on our plans and credits page.