- Every published win for automated prompt writing shares one property: there was a score to optimise against. Take the score away and a generator is rewriting, not improving.
- The APE paper searched candidate instructions against a scoring function and matched or beat human written instructions on 19 of 24 tasks, which is a search result rather than a writing result.
- Anthropic reports its prompt improver lifting multilabel classification accuracy by 30 percent on 500 Wikipedia articles and reaching 100 percent word count compliance on a summarisation task. Both targets are measurable, which is the point.
- A generator can fix structure, format and missing examples. It cannot know your refund policy, your product vocabulary or what a good answer looks like in your business.
- The cold start is the honest use case. When you do not know how to begin, a generated prompt is a better blank page than a blank page.
- A prompt has five parts. Once you can name them, you can repair a bad prompt in one line instead of regenerating the whole thing and losing the parts that worked.
The prompt generator pitch is appealing because the alternative sounds like a job. Learn prompt engineering, read the guides, practise. Or paste your rough idea into a box and get back something that looks professional, with sections and headings and a formal tone.
The looking professional part is real and it is also the trap. This piece works through where the generated version genuinely wins, where it quietly loses, and what a prompt is actually made of so you can tell the difference by reading rather than by trial and error.
Does automated prompt writing actually work?
Yes, under a condition that gets dropped when the research is summarised. The condition is that something was scoring the output.
The reference result is the APE paper from 2022, which treats an instruction as a program to be optimised. A model proposes many candidate instructions, each is scored by how well another model performs the task when following it, and the best one survives. Across 24 tasks, the automatically generated instructions matched or beat human written ones on 19 of them.
Read that mechanism again, because the popular summary of this paper is that models write better prompts than people. What the paper shows is that search beats writing when you have a scoring function. The generation step produced candidates. The scoring step did the work.
The commercial tools show the same shape. Anthropic's prompt improver reports a 30 percent accuracy gain on a multilabel classification task over 500 Wikipedia articles, and 100 percent compliance with a word count on a summarisation task across ten articles. Both of those have a right answer you can check by machine. Neither of them is write a description of my product that sounds like us.
Why does a generated prompt underperform on your specific task?
Because it optimises the half of the prompt that is generic and leaves the half that carries your business empty.
Take a real task. You want product descriptions for a catalogue of kitchen equipment sold to independent restaurants.
What a generator typically returns. A role line establishing an expert copywriter. A structured task statement. A tone instruction, usually engaging and persuasive. A list of sections to include, such as features and benefits. A word count. Sometimes an output format with headings. It reads well and it is longer than what you wrote.
What it cannot contain. That your buyers are chefs who care about the depth of a pan and the warranty, not about lifestyle. That you never claim dishwasher safe unless the manufacturer states it, because you were burned on a return. That your descriptions always mention the amperage because a restaurant kitchen has a fixed supply. That the word premium is banned in your catalogue because every competitor uses it.
Those four facts are worth more than every improvement the generator made, and no generator can produce them, because they are not properties of the task type. They are properties of your business, and the only place they exist is in your head.
This is the whole asymmetry. Structure is generic and can be generated. Substance is specific and cannot. A generated prompt starts you at a competent generic baseline, which is above where most people begin and permanently below where a specific one lands.
When is a generator the right move?
Three situations, and they have a common feature: you do not yet know enough to write the specific version.
| Situation | Generator | Why |
|---|---|---|
| Cold start on an unfamiliar task | Use it | A structured draft is a better starting point than an empty box |
| A domain you do not know | Use it | It surfaces the vocabulary and the sections a specialist would expect |
| Fifty variations of a working prompt | Use it | Volume with a fixed target is exactly the search problem it is good at |
| A task you do every week | Write it | You already hold the specifics, and generation dilutes them |
| Anything with your policy or tone in it | Write it | The constraints that matter are unguessable from outside |
| A prompt already working in production | Leave it | Regenerating loses the corrections you cannot remember making |
The last row deserves an extra sentence. A prompt that has been in use for months has accumulated repairs, each one made after something went wrong. Those repairs look like clutter to a generator, which will tidy them away. That is how a rewrite makes an output subtly worse in ways nobody can trace, and it is a specific instance of the general problem covered in how to tell whether a change actually helped.
What is a prompt actually made of?
Five parts. Naming them is the fastest way to stop needing a generator, because a bad output usually means exactly one part is missing and you can add that part in a line.
Who is speaking and to whom. Not a personality. The relationship. A supplier writing to a trade buyer produces different text from a brand writing to a consumer, and that one distinction changes more than any tone adjective you could add.
The task, in one sentence. If it takes two, it is two tasks and should be two prompts. Compound tasks are where outputs go vague, because the model splits its attention and does both adequately.
The constraints that are real. Length, banned words, claims you may not make, facts that must appear. Only include constraints you would actually reject an output for. A prompt full of nice to haves teaches the model that your instructions are decorative.
Two examples of good. This is the part people skip and it is the most powerful, because an example carries a hundred implicit rules you would never think to write down. Anthropic's prompt improver treats example enrichment as one of its five techniques, and it is doing for you what you could do better yourself, since your examples come from your catalogue and its come from nowhere.
The shape of the answer. Fields, sections, a row per item. Naming the shape is how you make an omission visible instead of invisible, and it is the same instinct that structured output enforces at the API level.
Anthropic's own prompting guidance offers a test that covers all five at once: show the prompt to a colleague who knows nothing about the task and ask them to follow it. If they hesitate, the model will too. The same documentation makes a point worth repeating to anyone who has been told to keep prompts short, which is that clarity beats brevity and a specific long prompt outperforms a vague short one.
What about the prompt templates circulating on social media?
Same problem, worse version. A generated prompt at least saw your rough description. A template saw nothing, and it was optimised for being screenshotted.
The tell is length combined with vagueness. A template that opens by declaring the model a world class expert with twenty years of experience, then asks it to think step by step, then demands it act as several personas at once, is performing seriousness rather than specifying anything. Read it as a colleague would: nothing in it tells you what to actually produce, which means nothing in it tells the model either.
There is a second, quieter cost. Templates train you to think of prompting as collecting incantations rather than as writing a brief. Someone who has collected forty templates often cannot fix a bad output, because they never learned which part of a prompt controls which part of the answer. Someone who understands the five parts can repair any template in a minute, including a good one.
The salvageable part of a template is its structure, occasionally its list of sections for a task you have never done. Take that, throw away the persona theatre, and fill in the specifics yourself. That is the same edit as the one for generated prompts, which is not a coincidence: both are generic scaffolding waiting for the content only you have.
Does any of this change with newer models?
The formatting tricks age badly. The structure does not.
Instructions that exist to work around a limitation tend to expire. Telling a model to think step by step mattered more when models did not do it by default, and on systems trained to reason natively the instruction can be redundant or actively unhelpful. Elaborate role play has been steadily losing value for the same reason. Anthropic maintains model specific guidance alongside its general principles precisely because this layer moves.
What has not moved in three years is the part that carries information: state the task once, give real constraints, show examples, name the output shape. Those are not model tricks. They are how you brief anyone, and they survive every release because they are about the content of the instruction rather than about the quirks of the reader.
Practical consequence for a small business: do not invest time in learning technique, invest it in writing down your specifics once. The specifics keep their value when the model changes. The technique does not, and a generator will happily supply the technique layer for free whenever you need it.
How should you use one, if you use one?
As a first draft you are going to overwrite, not as a finished artefact.
Generate. Then read it looking for the sentences that could have been written about any business in your sector, and replace each one with something only you could have written. The tone instruction becomes two banned words and one required phrase. The generic audience line becomes who actually buys. The features and benefits section becomes the three attributes your customers ask about on the phone.
Then delete anything left that you would not enforce. Generated prompts are long because length looks like effort, and every unenforced instruction dilutes the ones that matter. The OpenAI guidance on writing tool definitions makes the same point in a different context: do not make the model fill in things you already know, and keep the surface small enough to choose well.
What you end up with is usually shorter than the generated version and much longer than what you started with. That is the right shape. Our collection of prompt patterns with a before and after shows the same edit applied to five different tasks, and our note on what belongs in a system prompt covers where these instructions should live once the task is running regularly rather than being typed each time.
How do you score your own prompts without a benchmark?
You build the smallest possible version of what the research does, which takes about an hour and turns opinion into evidence.
Collect ten real inputs. Not invented ones. Ten actual products, ten actual customer emails, ten actual whatever the task consumes. Invented inputs are always tidier than reality and they will make every prompt look fine.
Write down what a pass means before you look at any output. Three or four checks, each answerable yes or no. Does it mention the material. Is it under ninety words. Does it avoid the banned words. Does it state the warranty. Deciding this afterwards is how you end up grading on vibes and preferring whichever output you read second.
Run both prompts over all ten inputs and count. Not read, count. Twenty outputs and four checks is eighty boxes, which is twenty minutes of dull work and the only thing that will actually tell you which prompt is better. Almost nobody does this, which is why almost every opinion about prompts is a preference.
The result is usually not what people expect. The generated prompt tends to win on the mechanical checks, since format compliance is what its techniques target, and lose on the checks that encode your knowledge. That split is the entire argument of this article, arriving as a number on your own data rather than as a claim in an article.
Keep the ten inputs and the checklist. When you change the prompt in three months, or when the model version moves underneath you, rerunning takes ten minutes and answers the question that otherwise costs a week of arguing.
The honest summary of the research
Automated prompt writing works when three conditions hold together: many candidates can be produced cheaply, each can be scored automatically, and the scoring reflects what you actually want.
Most business tasks fail the second and third conditions. There is no automatic score for whether a product description sounds like you, and the human who could judge it is the same person deciding whether to write the prompt by hand. In that situation the generator is producing variations nobody can rank, which is motion rather than progress.
Where all three conditions do hold, and they hold more often than people think, the payoff is real. Classification with labelled examples. Extraction where the fields are either right or wrong. Anything with a format you can check by machine. If your task looks like that, generate a batch, score them against your own data, and keep the winner. That is the actual technique behind every headline result in this field, and none of the consumer tools do the scoring step for you.
The same distinction shows up in how AI tools charge. Paying for volume makes sense when a machine can pick the winner, and paying for one careful output makes sense when only you can. Our pricing page shows how MaShop meters generation, which is worth understanding before you run fifty variations of anything.
What to do this week
Take the prompt you reuse most. Count how many of the five parts it contains. Most reused prompts have the task and nothing else, which is why they produce competent generic output and why a generator seems to improve them.
Add the missing parts by hand, starting with two real examples from your own work. Fifteen minutes, no tool involved. Then, if you still want to run it through a generator, you will have something to compare against, and you will be able to tell whether it improved the prompt or merely lengthened it.
That comparison is the skill. Once you have it, the question of whether to use a generator stops being interesting, because you can look at any prompt and see what it is missing.