Which Computer Vision Task Is Your Problem, Really?

You have a visual problem. Count the boxes on a pallet, spot the cracked tiles, read the supplier code off a label, sort incoming photos by product. Which of those is a classification problem and which is a segmentation problem, and does it matter?

It matters more than the choice of model, because the task type determines what you have to annotate, and annotation is where the money goes. Teams argue for weeks about architectures and then discover the labelling bill is ten times the compute bill. Getting the task type right is the highest leverage decision in the project and it happens in the first hour.

What are the main computer vision tasks?

Five, distinguished by the shape of what comes out rather than by how they work inside.

Classification returns one label for a whole image. Is this a defect or not. Which of my twelve product categories is this. The output is a word, sometimes with a confidence.

Object detection returns a box around each thing it found, with a label. This is what you need when the answer involves how many, or where, rather than what.

Semantic segmentation labels every pixel with a class. All the road pixels are road, all the crack pixels are crack. It does not separate one crack from another.

Instance segmentation does the same but keeps objects apart, so you get a mask per individual thing. This is what you need to measure the area of each separate item.

Keypoint estimation returns specific points, such as the corners of a document or the joints of a body. Text recognition returns the characters in the image, which is a distinct pipeline with its own conventions.

Which task does your problem actually need?

Ask what decision the output drives. Then choose the cheapest task that supports that decision, and no more.

This single question resolves most scoping arguments. If the outcome is that somebody inspects the item or they do not, a yes or no per image is sufficient, and that is classification. You have just avoided a segmentation project. If the outcome depends on how many items are present, you need detection, because a label cannot count. If the outcome depends on how much area is affected, or on a measurement in pixels, only then do you need masks.

The failure pattern is consistent. Somebody sees a demo where a model paints beautiful coloured masks over an image, decides that is what sophistication looks like, and scopes a segmentation project to answer a question that a single label would have answered. The masks are impressive. They also multiply the labelling cost and the annotation review burden, and the extra information is discarded downstream.

Why is annotation the real cost?

Because it scales with the number of examples and the fiddliness of each one, and neither goes away with a better model.

The numbers that exist are worth knowing. Research on faster box annotation reports that extreme clicking brought annotation down to 7 seconds per box, five times faster than the traditional way of drawing boxes, citing the figure that annotating a high quality box for ImageNet took about 35 seconds. The same work found the resulting boxes were as good as the original ground truth and that detectors trained on them were as accurate.

Hold that against the scale a real dataset needs. Microsoft COCO contains 328,000 images with 2.5 million labelled object instances across 91 categories, with objects labelled using per instance segmentations, built through extensive crowd worker involvement across separate interfaces for category detection, instance spotting and instance segmentation. Three interfaces, because the job splits into three different kinds of tedium.

For pixel level work the clearest evidence is what teams choose to build. The Cityscapes dataset ships 5000 images with fine annotations and 20000 with coarse ones across 30 classes. Nobody builds four times as much deliberately worse data unless the good data is expensive. The coarse set exists because fine annotation could not be afforded at that scale.

Tool vendors say the same thing structurally. Amazon's documentation for a semantic segmentation labelling task describes workers classifying pixels into predefined classes, notes that images containing large numbers of objects require more time, and ships an AI assisted auto segmentation tool specifically to help workers label them faster. Nobody builds an assistive tool for the cheap task.

What goes wrong with the data?

Three things, and all of them are visible before you train if you go looking.

The thing you care about is rare

Defect detection is the classic case. If one item in a hundred is faulty, a model that predicts "fine" every time scores 99 percent accuracy and has learned nothing. Accuracy is the wrong measure whenever the interesting class is rare, and it is almost always the rare class you built the system for.

Report precision and recall separately, and decide which one you are willing to trade. A quality inspection system that misses defects is worthless. One that flags too many wastes a person's day. Those are different failures with different costs, and a single accuracy number hides both.

Rare classes also need deliberate collection. If you have four hundred good photos and eleven defects, no model will learn the defect. Gathering more of the rare class is usually more effective than any modelling change, which is unglamorous and true.

The model was trained on a different world

A model trained on images from one camera, at one mounting height, under one set of lights, degrades when any of those change. Move the camera, replace a fluorescent tube with an LED, run it at a second site, and performance drops for reasons that have nothing to do with the model.

The defence is to build the evaluation set from the deployment environment before training anything. Photograph the real thing, in the real place, under the conditions that will exist, including the bad ones. That set is what tells you whether the project works, and collecting it first stops you from optimising against a fantasy.

The labels disagree with each other

Two annotators shown the same borderline item will label it differently, and that ceiling propagates into every score you compute afterwards. Have a second person relabel a sample and measure how often they agree. If humans agree only 85 percent of the time, a model scoring 85 percent has matched your data, not failed.

How do you evaluate a vision model without being misled?

Match the measure to the task, and never accept a single number.

For classification, look at the confusion matrix rather than accuracy. It shows which classes get mistaken for which, and the pattern usually points straight at a labelling problem or a missing category.

For detection, the standard measure combines how well predicted boxes overlap true boxes with how many objects were found at all. The important practical detail is that a detector scoring well on average can be poor on small objects specifically, which matters enormously if your objects are small. Ask for the breakdown by object size, not the headline figure.

For segmentation, overlap between predicted and true masks is the measure, and it punishes boundary errors heavily. A mask that is right in the middle and sloppy at the edges scores badly even when it answers your question perfectly, which is a reason to check whether you needed masks at all.

Across all of them, the discipline is the same one we applied to what a quoted word error rate hides: a benchmark number describes a pairing between a system and a dataset, and yours is not that dataset.

How many labelled images do you actually need?

Fewer than the literature implies, because you are almost certainly fine tuning rather than training from scratch.

The datasets quoted above exist to train general purpose models from nothing. Your project starts from one of those models and adapts it, which changes the arithmetic completely. For a classification task with clearly distinct categories, a few hundred images per class is a reasonable starting point, and the first useful signal often arrives well before that. For detection the unit is objects rather than images, so a few hundred instances of each class is the equivalent target, and one busy photograph can contribute many.

The honest answer is that nobody can tell you the number in advance, because it depends on how visually similar your classes are. Two categories that differ obviously need very little. Two that differ by a subtle texture need a lot. This is knowable cheaply: label a small set, train, look at the errors, and the curve tells you whether more data is the answer or whether the categories themselves are the problem.

Watch for the case where more data does not help. If performance plateaus well below what you need and the confusion matrix shows the same pair of classes swapping, you have a definition problem rather than a data problem. Two categories that your own annotators cannot separate reliably will not be separated by a model, and the fix is to merge them or to change what you photograph.

Deploy it as an assistant before you deploy it as a decision

The safest first deployment puts the model in front of a person rather than in place of one, and it pays for itself in evidence.

Run the system alongside the existing process for a few weeks, recording what it would have decided without acting on it. You get a measurement on live data, in the real environment, at zero risk, and you find out how often it disagrees with the human. Those disagreements are the most valuable training data you will ever collect, because they are concentrated exactly where the model is weak.

Then automate the confident cases and route the rest to a person. Most vision systems in working businesses run this way permanently rather than as a transitional stage, and it is a better design than full automation, not a compromise on the way to it.

Where does text recognition fit?

Alongside rather than inside, because the pipeline and the failure modes are different.

Reading characters from an image is a detection problem followed by a recognition problem, usually followed by a layout problem: knowing that this number is the invoice total rather than a line item requires structure, not just characters. That third stage is where most document projects actually fail, and it is not a vision problem at all.

If your visual problem is really a document problem, the scoping questions are different enough that we treated them separately in the walkthrough of building a document extraction pipeline. The short version: measure field level accuracy on the fields you care about, not character accuracy across the page.

Where should the model run?

Wherever the images are, if you can manage it, and vision is unusually well suited to that.

Vision models are often small compared with language models, and the input is bulky. Sending every frame to a server costs bandwidth and adds latency, while running a compact detector on a device next to the camera costs neither. For counting or inspection at a fixed location, on device is frequently both cheaper and more reliable, since it keeps working when the connection does not.

The trade is the one we set out in what actually fits on device, and the vision case sits at the favourable end of it.

What can a small business realistically do here?

More than it thinks for classification and text, less than it hopes for anything involving masks.

Sorting and tagging incoming photos is genuinely accessible. A few hundred labelled images per category, a fine tuned classifier, and you have automatic categorisation of product shots. This is the same kind of work as keeping a catalogue tidy, which is why it fits naturally beside the cataloguing decisions you already make when you build an online store with AI.

Reading text off labels, invoices and packing slips is accessible because pretrained systems handle the character recognition and your work is the structure on top.

Counting discrete items is accessible with detection if the items are separated and the camera is fixed. It becomes hard fast when items overlap, which is exactly the case people want it for. Stock counting from a photo of a full shelf is a harder problem than it looks, and worth prototyping before committing, alongside the more tractable parts of managing inventory with AI.

Anything requiring per pixel measurement of irregular shapes is a project rather than a feature. That is not a reason to avoid it, but it is a reason to scope it as one.

A scoping sequence that avoids the usual waste

Write down the decision the output drives, in one sentence, before naming any task type. Then pick the cheapest task that supports that sentence.

Collect a hundred images from the real environment, including the awkward ones, and label them yourself. Doing it personally is the point: an hour of labelling teaches you more about the problem than a week of reading, and it gives you a defensible estimate of what a full set will cost.

Check the base rate of the class you care about. If it is rare, plan the collection before the modelling.

Then try the zero training option first. If a general vision model answers your question adequately on those hundred images, you have finished. If it does not, you now have an evaluation set, a cost estimate and a clear statement of the gap, which is everything you need to brief somebody properly.

The short version

Vision problems come in a handful of shapes, and the shape you choose sets your labelling bill for the life of the project. Classification is cheap, detection is moderate, masks are expensive, and the impressive looking option is rarely the one your decision requires.

So start from the decision, not the demo. Build your evaluation set from the place the system will actually run. Look at precision and recall separately when the thing you care about is rare. Most vision projects that fail were mis scoped in the first hour, and that hour is free.