Running an LLM Locally: What Your Hardware Can Handle

What can this machine actually run? That is the question, and almost every answer you find online skips the arithmetic and jumps to a recommendation about which graphics card to buy.

The arithmetic is not complicated, and it changes the recommendation. Three separate things compete for the same memory, the third one is the one people forget, and the shortfall it causes is why a model that should fit runs at reading speed instead of thinking speed.

What actually decides whether a model fits?

Three claims on the same pool of memory, in descending order of size and ascending order of how often they get forgotten.

The weights come first, and their size is bits per weight multiplied by parameter count. This is the number everyone quotes. A model with 8 billion parameters at roughly 4.9 bits per weight occupies about 4.58 GiB on disk, according to the benchmark table in llama.cpp's quantize documentation, and it takes roughly that much memory when loaded.

The context window comes second, and it is variable rather than fixed. Every token you keep in the conversation occupies key and value cache memory, and that cost scales with how long the conversation runs. This is why a model that loads fine can run out of room forty messages in, and why the context figures printed on a model card are an architectural ceiling rather than a promise, a point we work through when comparing the current open weight families.

Runtime overhead comes third. The framework, the operating system, whatever else the machine is doing, and on a shared machine the display. Leave headroom or the system will make the decision for you by moving layers to the CPU.

That last outcome is the one to design against. Ollama's context length documentation advises using the maximum context a model supports while avoiding CPU offload, and tells you to check with a status command that shows 100 percent GPU rather than a split. A model half in system RAM does not run at half speed. It runs at a small fraction of it, because generation is bound by memory bandwidth and system RAM has an order of magnitude less of it than a graphics card.

Which models does your hardware really run?

This table pairs the memory guidance published for Ollama with the quantisation sizes from the llama.cpp benchmarks. Treat the VRAM column as the floor for a comfortable run, not the absolute minimum at which the model will load.

Two things to read out of that table. The 8B tier at 4-bit is where the value is concentrated, because it fits a card most people already own and the quality gap to 13B is smaller than the memory gap. And the jump from 13B to 30B is where cost stops being incremental, because you cross out of consumer graphics cards and into either a workstation card or two of something.

Why does a heavier quantisation run faster?

Because generation is limited by how fast weights move from memory, not by arithmetic. Smaller weights move faster. The llama.cpp benchmark table makes this concrete on Llama-3.1-8B, and the numbers are more dramatic than most people expect.

Generation more than doubles going from full precision to 4-bit, and the file shrinks to under a third. Prompt processing, meanwhile, barely moves: 923 tokens per second at F16 against 822 at Q4_K_M, which is within the range you would expect from run to run variation. Prompt processing is compute bound, so shrinking the weights does not help it.

That asymmetry has a practical consequence. If your workload is long prompts and short answers, summarising documents for instance, quantisation buys you file size and very little speed. If it is short prompts and long answers, quantisation is close to free performance. Knowing which one you have tells you whether to spend your remaining memory on a bigger model or a bigger context.

The quality cost is real and it is not linear. Community measurements on llama.cpp put Q4_K_M within a couple of percent of full precision perplexity, while 2-bit quantisation degrades sharply. Q4_K_M is the recommended default for a reason, and the 1-bit row in that table is in the benchmark because it is interesting, not because you should ship it.

How much quality does quantisation actually cost?

Less than you fear down to 4-bit, and much more than you expect below 3-bit. The curve is not a slope, it is a shelf followed by a drop.

The measurement usually quoted is perplexity, which is a rough proxy for how confidently a model predicts real text. Lower is better, and the useful part is the relative change rather than the absolute figure. Against a full precision baseline around 5.96, Q4_K_M lands near 6.06, a degradation of roughly 1.7 percent. Q2_K comes in near 6.87, about 15.3 percent worse, which is the point at which the community guidance stops recommending it for general use.

Put those two numbers next to the file sizes and the decision makes itself. Q4_K_M costs you under two percent of predictive quality and saves you nearly 70 percent of the file. Q2 saves a further chunk and costs you nine times as much quality. There is no interesting territory in between, which is why almost every published quantisation of a popular model ships a Q4_K_M variant and why it is the sensible default.

One caveat worth stating: perplexity is a proxy, not the thing you care about. It correlates with quality on general text and it says nothing specific about whether a quantised model still follows your instruction format or produces valid JSON. Structured output tends to degrade before prose does, so if your application depends on parseable responses, test that specifically rather than trusting a perplexity table.

Does the format matter as much as the size?

It decides what can run the file at all, which is a different kind of mattering.

The local ecosystem converged on GGUF, a binary format built for fast loading that packs the tensors and a standardised metadata block into one file. That metadata is the reason a runtime can open a model it has never seen and know the architecture, the vocabulary and the quantisation without a separate config. Hugging Face's documentation notes the contrast with tensor-only formats such as safetensors, which store the weights and leave the description elsewhere.

Practically, this means two things. First, if a model has a GGUF build published, you can run it with llama.cpp, Ollama, LM Studio or GPT4All without converting anything. Second, if it does not, converting is a real task involving a conversion script and then a quantisation pass, and it is the point at which a casual experiment becomes an afternoon.

Check for an existing GGUF before you commit to a model. Filtering the Hub by the GGUF tag takes ten seconds and it is the difference between downloading a file and building a toolchain.

What is the context trap?

That your VRAM decides your context window before you get a say, and the default is smaller than you think.

Ollama allocates context automatically based on available GPU memory. Under 24 GiB you get 4,000 tokens. Between 24 and 48 GiB you get 32,000. At 48 GiB or more you get 256,000. Those are the published tiers, and the first one is the one that catches people out, because 4,000 tokens is roughly a long email.

Now put that next to the other piece of guidance on the same page: tasks involving web search, agents or coding tools should run with at least 64,000 tokens of context. Read together, those two statements say something the hardware guides do not. If you want to run a local coding agent properly, the model size is not your binding constraint. The context is, and it puts you at 24 GiB minimum and realistically higher.

This is the single most useful thing to know before spending money. A 13B model on an 8 GB card is a perfectly good assistant for discrete tasks and a poor agent, not because it is not smart enough, but because it cannot hold the working context an agent generates. If agents are the goal, buy memory before you buy capability.

What do you give up against a hosted model?

Four things, and being honest about them makes the decision easier rather than harder.

• Ceiling. A quantised model that fits a consumer card is not competing with a frontier model on hard reasoning. It competes very well on the large volume of easy work, which is most work. We looked at how that gap has narrowed in our piece on how open-weight models now compare to frontier AI.
• Context. Covered above, and it is usually the binding limit rather than parameter count.
• Latency under load. One machine serves one conversation well. Serving several concurrently is a different engineering problem, which is what dedicated inference servers exist for, a tradeoff we broke down in our comparison of vLLM and Ollama.
• Maintenance. Model updates, driver updates, quantisation choices and the evening you will spend on a CUDA version mismatch. Hosted models have none of this and it is a real cost, just not one that appears on an invoice.

When does local stop being cheaper?

Later than most local enthusiasts admit, and the arithmetic is worth doing before the purchase rather than after.

Take a cheap hosted model as the comparison. Anthropic's published pricing puts Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens. Suppose you are a heavy individual user consuming 20 million input tokens and 4 million output tokens in a month, which is a lot of real work. That is $20 plus $20, so $40 a month.

Now let H be what you spend on hardware. Break even in months is H divided by 40, before electricity and before your time. At a hypothetical $2,000 machine that is 50 months. At $800 for a used card in a computer you already own, it is 20 months. Neither number is close to the payback period people imagine when they say local is cheaper.

The picture inverts under two conditions. The first is volume: a team of ten running the same workload turns $40 into $400 a month and pushes break even under six months on the same $2,000 box, because the hardware cost is fixed while the API cost scales with people. The second is workload shape, because bulk processing of documents overnight is exactly what an idle machine is good at, and it is the most expensive thing to buy per token.

Run your own version of that calculation with your own numbers before buying anything. If you are comparing against generating whole projects rather than raw tokens, our plans and credits breakdown shows the per project view of the same spend.

What does the setup actually involve?

Four steps, and the order matters because step two prevents the most common disappointment.

Check what you have. Find your VRAM figure before anything else, then read it off the table above. Everything downstream is decided by that one number, including which context tier you land in.

Pick the quantisation before the model. Q4_K_M is the right default. If the model you want does not fit at Q4_K_M, the answer is usually a smaller model at Q4_K_M rather than the same model at Q2. The quality curve falls off a cliff below 3-bit, and a well-quantised smaller model beats a badly quantised larger one on most real tasks.

Get the weights in the right format. Local runtimes read GGUF, a binary format built for fast loading that stores tensors and standardised metadata together. Hugging Face's documentation on GGUF explains the format and how to filter the Hub for models that ship in it, which is the practical way to find a quantisation someone has already produced rather than converting one yourself.

Verify it is on the GPU. Run the status command and confirm 100 percent GPU. If it shows a split, reduce the context, reduce the quantisation, or pick a smaller model. Do this before you form any opinion about local models being slow, because a partially offloaded model is not a fair test of anything.

When is local right regardless of cost?

Three situations, and in all three the money is not the argument.

When the data cannot leave. Regulated work, client confidentiality, personal health or financial records, or an internal policy that simply says no. A local model is not cheaper here. It is the only option that complies, and the break even calculation does not apply.

When you need it to work without a network. Field work, air-gapped environments, a laptop on a plane. Availability is the feature.

When the workload is bulk and unattended. Classifying a million records, extracting fields from a document archive, generating embeddings. These are the tasks where per token pricing is punishing and a machine running overnight for free is genuinely the right tool.

Outside those three, the honest answer is that a small hosted model is usually cheaper, always faster to start, and never your problem when a driver update breaks it. Local is worth doing anyway if you want to understand the stack, and that is a legitimate reason. It is just not a cost reason, and pretending otherwise is how people end up with an expensive card and a mild sense of grievance.

The short version

Count three things, not one: weights, context, overhead. Read your VRAM figure and let it choose your tier. Default to Q4_K_M and prefer a smaller model well quantised over a larger one crushed. If agents are the goal, memory beats parameters, because 4,000 tokens of context is the real ceiling on an 8 GB card and no model size fixes it. And do the break even sum with your own numbers, because for one person it usually lands years away, which means privacy, offline access and bulk throughput are the reasons that hold up.