- Whether a model fits on a device is arithmetic, not opinion. Parameters times bytes per parameter gives you the floor, and everything else is added on top of that number.
- At 4 bits per weight, a 7 billion parameter model needs about 3.5 gigabytes for weights alone, which is why reported real world figures land between 4 and 6 gigabytes once overhead is counted.
- The measured guidance on precision is unusually clear. Four bits is close to optimal for accuracy per bit, and quality falls away sharply below roughly 3.5 effective bits per weight.
- Most workloads that belong on a device are not language models at all. Wake word detection, vision classification and speech transcription fit comfortably where a chat assistant does not.
- The cost nobody budgets for is patching. A model behind an API is fixed the same hour. A model shipped to a fleet is fixed when users update, which for some devices is never.
Somebody has asked you to put the model on the device. The reasons given are usually privacy, latency or working offline, and all three are legitimate. The question that decides the project is narrower and it has an arithmetic answer: what will physically fit in the memory and the power envelope you have, and what has to stay in the cloud regardless of how much you would prefer otherwise?
Arm's definition of the field is a good place to anchor, because it is deliberately unglamorous. Edge AI is inference executed directly on devices at or near the data source, without relying on cloud connectivity, and edge devices face limits in compute, memory and storage that require compact models. That last clause is the entire engineering problem.
How do you work out whether a model fits?
Start with the weights, because they are the floor and they do not move. Memory for weights is roughly the parameter count multiplied by the bytes used per parameter. At 16 bit precision that is 2 bytes each. At 8 bits it is 1 byte. At 4 bits it is half a byte.
So a 7 billion parameter model at 4 bits needs about 3.5 gigabytes before anything else. A 3 billion parameter model at the same precision needs about 1.5 gigabytes. A 1 billion parameter model needs about half a gigabyte.
Then add what the arithmetic leaves out. The runtime itself, the tokenizer and vocabulary tables, activation memory during a forward pass, and the key value cache that grows with the length of the conversation. Practical reports for a 7 billion parameter model at 4 bits cluster between 4 and 6 gigabytes of working memory, which is the same figure the naive calculation gives once you allow for the parts it ignores.
That gap between 3.5 and 6 is where most on device projects go wrong. The model loads on the test device, the demo works, and then a real conversation grows the cache and the operating system reclaims memory from the application. Budget from the upper figure, not the lower one.
How low can precision go before quality breaks?
Lower than intuition suggests, and there is a measured answer rather than a rule of thumb. Dettmers and Zettlemoyer ran over 35,000 experiments across model families from 19 million to 176 billion parameters and published the result as The case for 4-bit precision: k-bit inference scaling laws. Their conclusion is that 4 bit precision is almost universally optimal for the tradeoff between total model bits and zero shot accuracy.
The framing matters as much as the number. They are not saying 4 bit models are as good as 16 bit ones. They are saying that if you have a fixed memory budget, you get more accuracy from a larger model at 4 bits than from a smaller model at higher precision. A 60 billion parameter model at 4 bits beats a 30 billion parameter model at 8 bits, at the same total bits.
A 2025 systematic evaluation of on device models across seven post training quantization methods, covering models from 0.5 to 14 billion parameters, reaches a compatible finding and adds a floor: heavily quantized large models consistently outperform smaller high precision models, with degradation accelerating below roughly 3.5 effective bits per weight. It also reports that resource use scales linearly with bits per weight, and that as models get smaller the throughput bottleneck shifts from moving data to raw computation.
The practical rule that falls out: quantize to 4 bits, choose the largest model that fits at that precision, and treat anything below about 3.5 effective bits as a research decision rather than a product one. Hugging Face's overview of quantization methods is the map of what is actually implemented, and the column worth reading first is not the bit width but the hardware support, because a method that cannot run on your target silicon is not a candidate whatever its compression ratio.
What fits on which class of device?
Four classes cover almost everything, and the useful column is the last one, because it converts a hardware spec into a decision.
| Device class | Usable memory for a model | Sustained power envelope | What realistically fits after 4 bit quantization |
|---|---|---|---|
| Microcontroller | Kilobytes to a few megabytes | Milliwatts, often on a battery measured in years | Keyword spotting, anomaly detection on a sensor stream, tiny classifiers. No language model of any size. |
| Phone or tablet | 1 to 4 GB in practice, since the operating system will not give you the headline RAM figure | A few watts in bursts, thermally limited within minutes | Vision models, on device speech, and language models roughly in the 1 to 3 billion parameter range. Seven billion is a flagship only proposition. |
| Laptop with an NPU | 4 to 16 GB | Tens of watts, sustained while plugged in | Language models up to roughly 14 billion parameters at 4 bits, plus real time vision and audio. |
| Edge server or gateway | 16 GB and upward, with a discrete accelerator | Hundreds of watts, mains powered, actively cooled | Most open weight models, several at once, serving a site rather than a user. |
Two clarifications that save arguments. A phone's advertised RAM is not your budget. The operating system reserves a large share and will terminate your process rather than let the system stall, so plan against what an application can hold under memory pressure rather than the number on the box. And a laptop NPU is not a small GPU: it is optimised for sustained low power inference on supported operations, which is a different shape of capability. Google's documentation on the LiteRT delegate for NPUs lists which vendor runtimes are wired up today and claims accelerated inference at lower energy than CPU or GPU execution, which is the real reason to target one.
Power is the constraint that gets discovered late. A phone can run a model at high speed for perhaps a minute before thermal limits reduce the clock, so a benchmark measured over ten seconds tells you almost nothing about a feature people use for ten minutes. Measure sustained throughput, not peak.
Which workloads belong on the device?
Sorting by workload rather than by device is the faster route to a decision, because the answer is stable across hardware generations while the memory numbers are not.
| Workload | Where it belongs | The honest reason |
|---|---|---|
| Wake word detection | On device, always | It must run continuously. Streaming audio to a server to detect a word is a privacy and bandwidth problem with no upside. |
| Image classification and object detection | On device | Small models are genuinely good at this, and video is expensive to upload but cheap to process locally. |
| Speech to text | On device for short utterances, cloud for long form accuracy | Compact speech models handle commands well. Transcribing an hour of multi speaker audio accurately still favours a larger model. |
| Short language assistance, such as reply suggestions or summarising a notification | On device, if you accept the quality ceiling | A 3 billion parameter model does this acceptably. It will not match a frontier model, and the feature has to be designed around that. |
| Long context reasoning, tool use, complex generation | Cloud | Memory alone rules it out, and the key value cache for a long context often exceeds the weights. |
| Anything needing knowledge newer than the shipped build | Cloud, or a hybrid | A device model knows what it knew at release. Retrieval can patch this, but the retrieval source is usually remote anyway. |
The row that changes projects is the last one. A great many features described as on device turn out to need current data, at which point the device is offline only in the sense that the model weights are local while the useful information is not. That is still a reasonable design, and it is worth being clear that it is what you are building.
What do you gain, and what do you actually give up?
The gains are the ones on every slide and they are real. Latency without a round trip, which for interactive features is the difference between a response and a wait. Data that never leaves the hardware, which changes what you must disclose and what a breach can expose. Function without a network, which for a vehicle or an industrial sensor is not a nicety.
The costs are less discussed and they compound.
You lose the ability to fix things quickly. This is the big one. A model behind an API is replaced in an afternoon and every user has the new one immediately. A model shipped inside an application is replaced when users update, which means a long tail running the old version indefinitely and a subset who never update at all. Every bug you ship is a bug you live with for a release cycle at minimum.
You inherit fleet variance. A cloud model runs on hardware you chose. A device model runs on everything your users own, with different accelerators, different available memory and different thermal behaviour. The same build will be fast on one phone and unusable on another, and you will not find out from a lab.
You ship the weights. Anything on a user's device can be extracted by a sufficiently motivated user. A 2026 systematic review of attacks and defences for on device AI inference makes the asymmetry concrete: roughly a quarter of the attack literature targets intellectual property theft while about half of the proposed defences address it, whereas adversarial attacks make up about a third of the attack literature with far less defensive work against them. Treat a shipped model as public, and put anything you cannot afford to disclose behind a network call.
You take on scale in the wrong direction. Cloud inference costs more as usage grows and less as you optimise. Device inference costs nothing per call and everything per supported device, which is a maintenance burden that grows with the number of hardware configurations rather than with revenue.
Why does the memory arithmetic keep being wrong in practice?
Because three things grow that the parameter calculation treats as fixed, and each one bites at a different point in the project.
The key value cache grows with the conversation. Weights are a constant. The cache that holds the attention state for the tokens already processed is not, and for a long exchange it can rival or exceed the weights themselves. A model that comfortably fits at the start of a session can push the process over the limit forty turns in. This is the failure that reaches production, because nobody tests a forty turn conversation on the oldest supported handset.
The runtime is not free. Loading a model means loading an inference engine, a tokenizer and its vocabulary tables, plus whatever the accelerator driver reserves. On a device with two gigabytes of usable headroom this is a meaningful share rather than a rounding error.
Quantization overhead varies by method. The evaluation above found resource use scaling linearly with bits per weight, but with power and memory differing by algorithm at the same nominal bit width. Two 4 bit builds of the same model are not interchangeable, and the block size and data type choices that made the difference in the scaling law work show up here as real megabytes.
The correction is procedural rather than clever. Measure peak resident memory on the lowest specification device you intend to support, during the longest interaction you intend to allow, after the device has been warm for ten minutes. Every one of those qualifiers exists because a project shipped without it.
Is a hybrid design a compromise or the right answer?
Usually the right answer, and the reason is that the two placements fail in different directions rather than one being generally better.
The pattern that works is a small local model handling the common, latency sensitive, privacy relevant path, with an escalation to a larger remote model for the cases it cannot handle. What makes this hard is not the routing. It is deciding what counts as a case it cannot handle, since a small model is generally poor at recognising the limits of its own competence and will answer confidently rather than escalate.
Two mechanisms work better than asking the model. Route on the input, using length, domain or the presence of a task type you already know is out of scope. Or route on the output, running a cheap check on the local result and escalating when it fails. Both are boring and both are more reliable than a self assessment.
The survey Network Edge Inference for Large Language Models is the reference to read next if you are building this seriously. It covers system architectures for edge deployment, model optimisation and the resource management and scheduling side that becomes the actual work once you are running inference across a fleet rather than on one machine.
How should you prototype this?
On a laptop first, always, because the iteration loop on real target hardware is slow enough to kill the exploration you need to do. Run the candidate models locally, measure quality on your own task, and only then port the winner. Our walkthrough of what your hardware can actually handle when running a model locally covers the memory arithmetic in the same terms as the calculation above, and the numbers transfer directly.
If your target is an edge server rather than a handset, the question stops being whether it fits and becomes how efficiently you serve it, which is a different discipline with different tools. We compared the two most common answers in vLLM against Ollama for serving a model, and the short version is that one is built for throughput across many requests and the other for convenience on one machine.
Model choice deserves more care than it usually gets, because the gap between an open weight model and a frontier model narrows considerably at the small end where you are forced to operate anyway. We looked at where that gap actually sits in how open weight models compare with frontier systems on cost and capability. And because shipping weights and shipping data are different risks with different obligations, it is worth checking what your deployment commits you to before the architecture is fixed; our security page sets out how we draw that line for data that stays with the merchant.
The question to answer first
Not which model. Not which framework. This one: if this model turns out to have a serious flaw three months after launch, how long until every user has the fix?
If the answer is a few hours, you are describing a cloud deployment and the constraints in this article are optional. If the answer is a release cycle, or never for some devices, then on device is still the right call for the right reasons, and you now know the real price. Everything else here is arithmetic, and arithmetic is the easy part.