Run LLMs Locally: A Complete Ollama Guide

You do not need a cloud account or a monthly bill to use a capable language model anymore. If you want to run LLMs locally, on your own laptop or workstation, the open-source tool Ollama has become the most direct way to do it. It downloads open-weight models, serves them behind a clean API, and gets out of your way. This guide walks through what Ollama is, how to install it, how to pick a model your machine can actually handle, and how to build real applications against it, including tool calling and structured JSON output.

The appeal is simple. A model running on your hardware costs nothing per token, works offline, and never sends your prompts to a third party. For developers handling sensitive data, for anyone on a tight budget, and for people who just want to understand how these systems behave, local inference has gone from a hobbyist curiosity to a genuinely practical default. The models are good enough now that the gap with hosted frontier systems, while real, is small enough to ignore for many everyday tasks.

Why run LLMs locally in the first place

Before the how, the why. Running a model on your own machine solves several problems that hosted APIs create. Privacy is the obvious one. When inference happens on your hardware, no prompt or document leaves the building, which matters for legal, medical, or proprietary work where sending text to an outside vendor is a compliance headache. Cost is the second. Hosted APIs bill per token, and a heavy user or an automated pipeline can run up a real invoice, while a local model burns only electricity once the hardware is paid for.

Then there is reliability and control. A local model does not rate-limit you, does not deprecate the version you depend on, and does not go down because a provider had an outage. You choose the exact model and keep it as long as you like. For learning, that control is priceless: you can inspect behavior, swap models freely, and experiment without watching a meter. The trade-off is that you supply the compute and the quality ceiling is set by what your hardware can run, which is the honest constraint this guide keeps returning to.

What Ollama actually is

Ollama is often described as Docker for language models, and the analogy holds. It is an open-source platform that pulls open-weight models from a registry, stores them locally, and runs them on demand. Under the hood it is a Go web server that manages model loading and, per the project's own documentation, uses llama.cpp to perform the actual inference, spawning a llama.cpp process to serve each model. You never touch that machinery directly. You issue one command and get a running model with an API in front of it.

That design is why Ollama won mindshare over rolling your own llama.cpp build. Downloading model weights, converting them, choosing a quantization, and wiring up a server used to be a weekend project. Ollama collapses it into a single install and a single run command. Its model library, browsable at ollama.com/library, carries families such as Meta's Llama, Alibaba's Qwen and DeepSeek's reasoning models, alongside Gemma, Mistral and the open gpt-oss releases, among many others. Recent versions even fold in a launch command that wires local or cloud models into coding tools, so the surface keeps growing.

Installing Ollama and running your first model

Installation is deliberately boring, which is a compliment. On macOS or Linux you can run the official script, curl -fsSL https://ollama.com/install.sh | sh, and on Windows the PowerShell equivalent, irm https://ollama.com/install.ps1 | iex. There is a desktop app for macOS and Windows if you prefer a click, and an official ollama/ollama Docker image for servers and containers. The GitHub README documents all of these paths.

Once installed, one command gets you talking to a model. Type ollama run gemma4 and Ollama downloads the model if you do not have it, loads it into memory, and drops you into an interactive chat. That is the whole first-run experience. The commands you will use most are a short list. ollama pull downloads a model without running it. ollama run starts a chat or single prompt. ollama list shows what you have downloaded. ollama ps shows what is currently loaded in memory. ollama rm deletes a model to reclaim disk space. The full set lives in the CLI reference.

Behind the interactive session, Ollama is already running a server. By default it listens on port 11434, and ollama serve starts that server explicitly if it is not already up. Everything else in this guide talks to that server, whether you use the chat window, the command line, or code.

Picking a model your hardware can handle

This is where local inference gets real. A model's memory footprint scales with its parameter count and its quantization, which is the technique of storing weights at lower precision to shrink them. A rough industry rule that Ollama's own materials have long echoed: plan on at least 8 GB of RAM for 7-billion-parameter models, 16 GB for models around 13 billion parameters, and 32 GB for models near 33 billion. Larger models climb from there, and running them well usually means a GPU with enough video memory to hold the weights.

Quantization is the lever that makes this workable. A model quantized to 4-bit precision uses roughly a quarter of the memory of its full 16-bit form, with a modest quality cost that is often imperceptible for chat and coding. Most models in Ollama's library ship in quantized form by default, which is why a 7-billion-parameter model can run comfortably on a mainstream laptop. The practical advice is to start small, confirm the speed is acceptable, and only move up in size if your hardware has headroom. A model that swaps to disk because it does not fit in memory will feel unbearably slow, so fitting the weights in RAM or VRAM matters more than raw model size.

The OpenAI-compatible API and building real apps

The feature that makes Ollama a serious development tool is its API. It exposes native REST endpoints, most importantly /api/generate for single completions and /api/chat for multi-turn conversations, both documented in the API reference. A minimal chat request is a curl call to http://localhost:11434/api/chat with a model name and a list of messages, and the server streams back a response.

The bigger convenience is the OpenAI-compatible layer. Ollama mirrors OpenAI's Chat Completions format at http://localhost:11434/v1, which means tools and libraries written for OpenAI usually work against a local model with almost no change. In the official OpenAI Python or JavaScript client you override the base URL to point at Ollama and set the model name to a local one, and existing code keeps working. That one-line switch is what lets a team prototype against a hosted model and later move the same application to local inference, or run both side by side. It removes the usual rewrite that switching providers demands.

Structured outputs: getting reliable JSON

Anyone who has tried to parse an LLM's free text into a program knows the pain of a model that mostly returns valid JSON. Ollama addresses this with structured outputs, which constrain a model's response to a format you define with a JSON schema. You pass the schema in a format parameter, and the model is forced to emit output that conforms to it. The feature is explained in Ollama's structured outputs post, and it changes JSON from a hopeful request into a guarantee.

In practice the ergonomics are pleasant. Python developers can define the shape with a Pydantic model and hand its generated JSON schema to the format parameter, while JavaScript developers can do the same with Zod. Ollama recommends this typed approach because it keeps the schema and your code in sync. A few habits improve results. Set the temperature to zero so the model is deterministic rather than creative when filling fields. Keep the schema shallow, since deeply nested or recursive structures can confuse the constraint engine. Mark fields optional when a value might genuinely be missing, so the model is not forced to invent one. With those in place, structured extraction becomes reliable enough to sit inside a production pipeline.

Tool calling and building agents

Beyond text and JSON, Ollama supports function calling, the mechanism that lets a model decide to invoke a tool you have defined and pass it arguments. This is the foundation of agentic behavior: a model that can call a weather function, query a database, or hit a search API rather than guessing. Function calling works with models trained for it, a group that includes recent Llama releases and other capable open-weight models, and it streams in real time so the tool call surfaces as the response is generated.

The pattern mirrors what hosted providers offer. You describe the available tools and their parameters, send them with the prompt, and the model responds either with an answer or with a request to call a specific tool. Your code runs the tool, feeds the result back, and the model continues. Because this rides on the same OpenAI-compatible surface, agent frameworks built for hosted APIs frequently run against Ollama unchanged. The limiting factor is model quality: smaller local models are less reliable at choosing tools correctly than the largest hosted systems, so agents built on modest hardware need tighter prompts and more validation.

Embeddings and local retrieval

Ollama also generates embeddings, the numeric vectors that represent text for semantic search. This is the backbone of retrieval-augmented generation, where a system finds relevant documents and feeds them to the model as context. Running an embedding model locally means you can build a private search or question-answering system over your own files without any of that content touching an external service. You pull an embedding model, send text to the embeddings endpoint, and store the resulting vectors in a database of your choosing.

Pairing local embeddings with a local chat model gives you a fully self-contained RAG stack. A document is embedded, stored, retrieved by similarity when a question comes in, and passed to the model as grounding. For sensitive corpuses, legal archives, internal wikis, or personal notes, this keeps the entire loop on your machine. The quality depends on the embedding model and your chunking strategy, but the privacy guarantee is absolute in a way no hosted pipeline can match.

Customizing models with a Modelfile

Ollama lets you build your own model variants with a Modelfile, a small text file reminiscent of a Dockerfile. In it you name a base model, bake in a system prompt, and set parameters like temperature or context length. Running ollama create against that file produces a named model you can run like any other. This is how you turn a general base model into a specialized assistant, say a terse code reviewer or a customer-support persona, without touching the weights themselves.

The Modelfile approach keeps configuration versionable and shareable. Instead of remembering a pile of runtime flags, you capture the setup once and distribute the file. A teammate runs the same create command and gets an identical model. It is a modest feature, but it is the difference between a one-off experiment and a reproducible tool you can rely on across a team.

Limits and gotchas worth knowing

Local inference is not free of trade-offs, and pretending otherwise sets you up for frustration. The first limit is quality. The best open-weight models are strong, but the very largest hosted systems still lead on the hardest reasoning and coding tasks, and you feel that gap most on complex multi-step problems. For summarization, drafting, extraction, and routine coding help, a good local model is more than adequate. For frontier-grade reasoning, it may not be.

The second limit is hardware, which sets a hard ceiling you cannot prompt your way past. If a model does not fit in memory it will crawl, and a machine without a capable GPU will run large models slowly even when they fit. Context length is another quiet constraint: longer context windows consume more memory, so the effective window you can use locally may be smaller than the model's maximum. And while quantization is a gift, aggressive quantization does degrade quality, so a heavily compressed model can feel noticeably duller than its full version. Knowing these edges lets you match the tool to the task instead of expecting a laptop to behave like a data center.

Hardware acceleration is worth understanding because it decides how usable local inference feels. On Apple Silicon, Ollama uses an optimized engine that takes advantage of the unified memory shared between the CPU and GPU, which is why recent Mac laptops punch above their weight for their price. On desktops, an Nvidia GPU with generous video memory delivers the fastest experience, since the whole model can live on the card, and Ollama also offers acceleration on AMD graphics through preview support on Windows and Linux. If a model runs only on the CPU it will still work, just slowly, so investing in memory and a capable GPU is the single biggest speedup available to a local setup. The gap between a model that fits on the GPU and one that spills to system RAM is often the difference between a snappy assistant and one you abandon after a day.

Where local inference fits in a real workflow

The honest way to think about Ollama is as one tier in a spectrum, not a wholesale replacement for hosted AI. Many developers land on a hybrid pattern: local models for private data, rapid iteration, and high-volume grunt work, with a hosted frontier model reserved for the occasional task that genuinely needs the extra capability. Because Ollama speaks the OpenAI protocol, moving a request from one tier to the other is a configuration change rather than a rewrite, which makes that hybrid genuinely easy to run.

For anyone starting out, the path is short. Install Ollama, pull a small well-regarded model, and talk to it from the command line to get a feel for the speed on your hardware. Then point a script at the local API and confirm your existing code works with a changed base URL. Add structured outputs when you need reliable JSON, function calling when you need tools, and embeddings when you need retrieval. Each step builds on the same server you started in minute one. The result is a private, no-cost, always-available AI stack that lives entirely on hardware you control, and that is a meaningfully different footing than renting every token from someone else. If you already build against hosted APIs, the smartest next move is to run the same workload locally once and see how little has to change.