Run Local LLMs with Ollama: A Practical Setup Guide

You do not need a cloud account, an API key, or a monthly bill to run local LLMs on your own machine. Ollama has become the default way to do it, turning a laptop or a spare desktop into a private AI server with a single command. It downloads models for you, handles the fiddly parts like quantization and GPU acceleration, then exposes a clean local API that other tools can talk to. This guide walks through installing Ollama, pulling your first model, managing what you have, calling the built in API, plus building custom model configurations, so by the end you can run capable open models entirely offline.

What Ollama is and why run local LLMs

Ollama is an open source tool that downloads, runs, plus manages large language models on your own computer. The common description is that it works like Docker for AI models. You pull a model with one command and Ollama takes care of the messy details underneath: choosing a quantized version that fits your memory, loading it efficiently, then using your GPU if you have a supported one. First released in 2023, it now runs on macOS, Windows, plus Linux, and it has become the on ramp most people use for private model hosting.

The reasons to run local LLMs are practical rather than ideological. The first is privacy. When the model runs on your machine, your prompts and documents never leave it, which matters for sensitive work, regulated data, or anything you would rather not hand to a hosted provider. The second is cost. There are no per token charges and no usage caps once the model is downloaded, so you can experiment as much as you like for the price of electricity. The third is independence. A local model keeps working with no internet connection at all, which is useful on a plane, in a secure environment, or during an outage. Those advantages come with an honest tradeoff. The largest frontier models still need data center hardware, so what you run locally will be smaller than the biggest hosted systems. For a huge range of everyday tasks, though, a good open model on your own machine is more than enough.

Installing Ollama on macOS, Windows, and Linux

Getting Ollama onto your machine takes a couple of minutes. The simplest path for any platform is the official download page at ollama.com/download, which offers a native installer for macOS and a Windows installer package. Run it, and Ollama installs as a background service plus a command line tool.

On Linux, or on macOS if you prefer the terminal, there is a one line installer. Running the command curl -fsSL https://ollama.com/install.sh | sh downloads the Ollama binary, sets it up as a systemd service on Linux, then adds the ollama command to your path. There is also an official Docker image for anyone who wants to run it inside a container. Whichever route you take, you can confirm the install worked by running ollama --version in a terminal, which prints the installed version number. Once installed, the Ollama service runs quietly in the background, ready to load a model the moment you ask for one. The full walkthrough lives in the Ollama quickstart.

Running your first local LLM

This is the part that surprises people with how little it takes. To run a model, type ollama run followed by the model name. For example, ollama run gemma4 pulls Google's Gemma model if you do not already have it, then drops you straight into an interactive chat prompt. The first run downloads the model weights, which can be a few gigabytes, so give it a moment. After that the model is cached locally and starts instantly.

Once you are at the chat prompt, type a message and press enter to get a response generated entirely on your hardware. The interactive session has a few built in commands that start with a slash. Typing /bye exits the session, and /set system lets you give the model a persistent instruction about how it should behave for the rest of the conversation. If you would rather just fetch a model without chatting, ollama pull followed by the model name downloads it in the background so it is ready for later use. The separation is handy. Use pull when you want to preload several models, then use run when you actually want to talk to one.

Managing the models you run locally

As you collect a few models, you will want to see and prune them. Ollama has short commands for this. Running ollama list shows every model you have downloaded along with its size on disk, which is the fastest way to see what is eating your storage. Running ollama ps shows which models are currently loaded into memory and running, useful when you are juggling several. When you are done with a model, ollama rm followed by its name deletes it and frees the disk space. For a closer look at a specific model, ollama show followed by its name prints its details, including the parameters and the template it uses.

Behind all of these sits ollama serve, which starts the Ollama server if it is not already running. Most of the time the background service handles this for you, so you rarely call serve directly. It matters mainly when you want to run the server manually, for instance inside a script or a custom environment where the automatic service is not present. Together these commands give you a simple lifecycle for local models: pull them, list them, run them, then remove the ones you no longer need.

Using the local API to run LLMs in your own apps

Chatting in a terminal is only the start. The real power of Ollama is that it runs a local web server your own software can call. By default that server listens at http://localhost:11434, and its REST API is what turns Ollama from a toy into infrastructure. The API documentation covers the details.

The core endpoint is /api/generate for a single prompt and response. A minimal call looks like this: curl http://localhost:11434/api/generate -d with a small JSON body naming the model and the prompt, for example a body of model set to gemma4 plus a prompt asking why the sky is blue. There is also an /api/chat endpoint for multi turn conversations that take a list of messages, which is what you want for anything resembling an assistant. By default the API streams its response token by token as the model produces it, which feels responsive in a UI. If you would rather receive the whole answer in one piece, you set stream to false in the request body.

One detail makes Ollama especially easy to adopt: it also exposes an OpenAI compatible endpoint. That means many tools written for the OpenAI API can be pointed at your local Ollama server by changing the base URL to localhost, with little or no other code change. Combined with the official Python and JavaScript libraries, this is what lets people swap a hosted model for a local one in an existing app. You keep your code and swap the engine underneath, which is the same portability argument that makes open models attractive in the first place. If you want a sense of which open models are worth running this way, our guide to the Mistral AI model lineup covers several that ship in sizes suited to local hardware.

Connecting Ollama to your favorite tools

Because Ollama speaks a standard local API, a whole ecosystem has grown up around it, and you rarely have to build a frontend yourself. The most popular companion is Open WebUI, a self hosted chat interface that gives you a browser based experience much like a hosted assistant, but pointed at your local models. It handles conversation history, multiple models, plus document upload, all running against the Ollama server on your own machine.

Developers who live in an editor tend to reach for extensions instead. Tools like the Continue extension for popular code editors let you use a local Ollama model as a coding assistant right inside your workflow, so autocomplete and chat about your code happen without sending that code to a cloud service. On the programming side, the major orchestration libraries such as LangChain and LlamaIndex include first class Ollama support, which makes it straightforward to build a retrieval augmented generation pipeline where your documents are indexed locally and answered by a local model. Because Ollama also exposes the OpenAI compatible endpoint mentioned earlier, many applications that were written against a hosted API can be redirected to your machine by changing a single base URL. The practical upshot is that adopting Ollama rarely means starting from scratch. You bolt it onto tools you already use.

Common problems when you run LLMs locally

A few snags come up often enough to be worth naming. If a command fails to reach the server, the usual cause is that the Ollama service is not running or that something else has claimed port 11434, so checking whether the port is free is the first move. If a model loads slowly or the machine grinds to a halt, you have likely picked a model too large for your available memory, and dropping to a smaller size or a more aggressive quantization fixes it. If inference feels sluggish even on a small model, the model may be running on the processor because a compatible GPU was not detected, which is common on machines with unsupported graphics hardware. None of these are hard to resolve once you know where to look, and the quickstart plus the model library between them answer most first day questions.

Customizing a model with a Modelfile

Sometimes you want a model that always behaves a certain way without repeating the setup each time. Ollama handles this with a Modelfile, a short text file that defines a customized model on top of an existing one. The concept mirrors a Dockerfile. You start from a base and layer your changes.

A Modelfile has a few key instructions. FROM names the base model you are building on, such as an existing Gemma or Llama model you have pulled. PARAMETER sets generation options like temperature, which controls how creative or deterministic the output is. SYSTEM defines a permanent system prompt baked into the model, so it always adopts a given persona or rule set. Once your Modelfile is written, the command ollama create followed by a name for your model and the flag pointing at the file builds it. From then on you run your custom model by its new name exactly like any other, and it carries your settings with it. This is how teams turn a general base model into a purpose built assistant, a support agent tuned to a friendly tone, or a coding helper with a fixed style, all without touching the underlying weights.

Choosing a model and the hardware to run LLMs

The most common beginner question is which model to run, and the honest answer is that it comes down mostly to memory. Model size is measured in parameters, usually written with a B for billion. A rough guide holds up well. Models in the 7B to 8B range are the sweet spot for most laptops and modest desktops, balancing quality against hardware demands, and current open options like Llama, Qwen, plus Gemma all offer strong models at that size. Step up to 13B or larger and you get better answers at the cost of needing noticeably more memory. The very large models in the tens of billions of parameters or beyond really want a dedicated GPU with a lot of memory, or a machine with generous unified memory.

Quantization is the trick that makes local models practical. It compresses a model's numbers to a smaller precision, shrinking how much memory it needs to load while keeping most of its quality. Ollama applies a sensible quantization by default, which is why a model that would nominally need a huge amount of memory can often run on a normal machine. A useful rule of thumb from popular setup guides is that a quantized 7B model runs comfortably on a computer with about 8 gigabytes of memory, a 13B model wants roughly twice that, plus the largest models scale up from there. Ollama also detects a supported GPU automatically and uses it, falling back to the processor if none is present. Running on the processor alone works, it is just slower, so a graphics card makes the experience much snappier for bigger models.

The practical way to start is to pick a 7B or 8B model, run it, then judge the speed and quality on your own tasks before reaching for anything larger. It is far easier to move up when you hit a real limit than to fight a model that is too big for your machine from day one.

From first command to a private AI setup

The reason Ollama caught on is that it collapses a genuinely complex task, hosting and serving a language model, into a handful of memorable commands. You install it once, pull a model, run it, then optionally wire its local API into your own tools or shape its behavior with a Modelfile. Everything happens on hardware you control, with prompts that never leave your machine and no meter running in the background. That combination of privacy, zero marginal cost, plus offline capability is why running local LLMs has moved from a niche hobby to a normal part of many developers' toolkits. Start small with a 7B model to learn the workflow, lean on the model library to match sizes to your hardware, then grow into the API and custom Modelfiles as your needs get more serious. The barrier that once kept local AI in the hands of specialists is mostly gone, and the only real way to appreciate that is to open a terminal and run your first model. A single command stands between you and a private assistant that answers on your own hardware, and that is a genuinely new thing to have on an ordinary computer.