Running open-weight AI models on your own computer, for free

Learn in your browser · Chapter 01 of 05

Before you install anything, it helps to know what you are choosing, and what 'free' does and does not mean here.

A hosted AI assistant such as a chatbot in your browser sends your text to someone else's server and charges you, directly or through a plan, for the privilege. An open-weight model is different: its parameters are published for anyone to download, so you can run it on hardware you already control. Free tools such as Ollama, LM Studio and llama.cpp load those weights and let you talk to the model without sending anything over the internet, once the model is downloaded.

Scroll sideways to see every column.

One more distinction matters before you start. Open-weight means the parameters are downloadable under some licence. Open source, in the stricter sense used by the Open Source Initiative's Open Source AI Definition, also asks for the training code, real information about the training data, and freedom to use the model for any purpose. Most of the well-known 'open' models you will meet in this workbook are open-weight, not fully open source by that definition. It rarely changes whether you can try a model on your own machine for personal use, but it matters if you plan to redistribute weights or use them commercially, which Open model licences, datasets and deployment obligations covers properly.

Learn in your browser · Chapter 02 of 05

Most local-model tools are a friendly front end over a much smaller number of inference engines. Get to know the engine and the front ends make more sense.

llama.cpp is a free, open-source engine, written in C/C++ and MIT licensed, that runs language models on ordinary hardware, including CPU-only machines and mixed CPU-and-GPU setups. It popularised the GGUF file format, a single file holding a model's weights plus the metadata needed to run it. A large part of the local-model ecosystem, including both tools below, builds on llama.cpp or a close relative of it. Each fact in this chapter was checked against the named tool's own documentation in September 2026; tools change quickly, so check the current page before you rely on a detail.

Scroll sideways to see every column.

Ollama. Install it from ollama.com, a one-line installer script for Mac and Linux, and a normal installer for Windows, then run a model by name. It downloads the weights the first time and reuses them afterwards.

# Install (macOS or Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a small model, then chat in the terminal
# (model names change: check ollama.com/library for the current list)
ollama run llama3.2

# See what you have downloaded
ollama list

# Remove a model you no longer want
ollama rm llama3.2

You can also point Ollama at a GGUF file you downloaded yourself, using a small text file called a Modelfile.

# Modelfile
FROM ./my-model.gguf

Save that next to the .gguf file as Modelfile, then run ollama create my-model -f Modelfile, followed by ollama run my-model.

LM Studio. Download the installer for Mac, Windows or Linux from lmstudio.ai. Inside the app, press Ctrl+2 (Cmd+2 on Mac) to jump to the Discover tab and search Hugging Face's model catalogue by name, pick a file, and download it with a click; LM Studio highlights the quantisation levels your hardware can realistically run. Load the model, then chat in the built-in window. LM Studio also ships a command-line tool called lms, and a local server compatible with the request format several hosted chat APIs use.

# Search Hugging Face from the terminal instead of the app
lms get qwen

# Download a specific model and quantisation from a Hugging Face URL
lms get https://huggingface.co/<org>/<model>-GGUF@Q4_K_M

llama.cpp directly. This route suits people who want to script things, run on unusual hardware, or avoid an extra layer of software. Download a prebuilt release or build it from source, then run its included server against a GGUF file. Exact binary names and flags change between releases, so read the current README rather than trusting an old example, including this one.

# Illustrative only: check the current README for the exact binary name and flags
./server -m my-model.gguf --port 8080

Learn in your browser · Chapter 03 of 05

You do not need the exact formulas to get started. You need one honest rule of thumb, and to know what quantisation is actually trading away.

A model's file size, and roughly the memory it needs to run, comes from two numbers: how many parameters it has, and how many bytes each one is stored in, its precision. The rule is simple: size in bytes is roughly parameters times bytes per parameter. Real quantised files are usually a little larger than this simple sum, because each block of weights also stores a small scaling value, so always trust the file size shown on the download page over your own arithmetic.

Scroll sideways to see every column.

Quantisation is the act of storing a model at lower precision than it was trained in, to shrink the file and the memory it needs. In the GGUF world you will see names such as Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K and Q2_K, plus a newer 'IQ' family, such as IQ4_XS or IQ3_S, for very low bit-widths. The number is roughly the bits used per weight; the letters describe the method, and K-quants generally mix precisions across a model's different parts, spending more bits where it costs more to be wrong. As a starting point, Q4_K_M or Q5_K_M is a common, reasonable default: noticeably smaller than the original, with a quality loss most people cannot easily spot in ordinary use. Go lower only when you are genuinely short of memory, and expect the loss to become more noticeable as you do.

Memory also comes in different places, and they are not interchangeable. VRAM is the memory on a dedicated graphics card: fast, and usually the smallest pool in the system, from a few gigabytes on older laptops to 24 GB or more on high-end cards. System RAM is larger and shared with everything else your computer is doing; running a model from RAM alone, on the CPU, works but is generally slower than running it on a GPU. Unified memory, used by Apple Silicon Macs, is shared between the CPU and GPU in one pool, so a Mac with a large amount of unified memory, and current configurations start at 16 GB and run up to 512 GB on the highest Mac Studio option, can often run larger models than a laptop with a similarly sized but separate VRAM pool, because the GPU can reach the whole pool.

Scroll sideways to see every column.

Treat that table as a rough starting point, not a promise: it assumes 4-bit weights, a modest context length, and leaves headroom for your operating system and everything else you have open. For the exact method, including the key-value cache that context length adds on top of the weights, see GPU memory maths for LLMs and SLMs.

Learn in your browser · Chapter 04 of 05

Hugging Face is the model hosting site both Ollama and LM Studio point to, directly or indirectly, and the most direct way to see what is actually available today.

Hugging Face hosts models from individual developers, universities and the companies that train frontier-scale models, searchable and filterable by task, format and licence. You do not need an account to browse or download public models, though creating a free one makes downloading and returning to models easier.

1. Go to huggingface.co/models.
2. Filter by task; for a chat model choose 'Text Generation'.
3. Filter by library or format: choose GGUF for llama.cpp, Ollama and LM Studio.
4. Sort by 'Trending' or by downloads, to see what people are actually using right now, not just what is newest.
5. Open a model's page and read its model card before downloading anything.

A model card is the closest thing to a label on the tin. Before you download, check: the licence, a name such as Apache-2.0 or MIT, or a custom licence you should actually read; the base model, if it is a fine-tune of something else; whether GGUF files are available, often in a separate repository named with '-GGUF' at the end, sometimes uploaded by the community rather than the original publisher; and any stated intended use or limitations. Prefer safetensors or GGUF files over an older pickle-based format: Hugging Face's own documentation warns that pickle files can execute code when loaded, which is a real security risk for a file you downloaded from a stranger.

Scroll sideways to see every column.

Learn in your browser · Chapter 05 of 05

Time to put it together: download, load, prompt, and know what to check when something goes wrong.

1. Pick a tool, Ollama or LM Studio, and a model and quantisation, using chapters 1 to 3.
2. Download it. This can take a while on a slow connection; a 4-bit 7 to 8 billion parameter model is typically a few gigabytes.
3. Load it, and send a first prompt drawn from your real task.
4. Watch two things: how long before the first word appears, and roughly how fast the words then stream.
5. If it is too slow or will not load, use the table below before giving up on the model entirely.

Scroll sideways to see every column.

What local models are honestly good at, on ordinary hardware: drafting and editing text, summarising something you already have in front of you, answering questions about your own documents, simple coding help, and anything where working offline or keeping data off someone else's server matters more than having the single most capable model available. What they are honestly weaker at, compared with the largest hosted assistants: very long or complex reasoning chains, broad general knowledge at the edges of what a much larger model was trained on, and tasks needing tools or web access the model itself does not have. Neither of those is a flaw in the tools; it follows directly from running something small enough to fit on your machine.

You now have a working local setup: a tool, a model that fits your machine, and a sense of what to check when it misbehaves. From here, GPU memory maths for LLMs and SLMs gives you the exact formulas behind chapter 2, Inference engines explained goes deeper on speed, batching and serving a model safely to more than one user, and Open model licences, datasets and deployment obligations covers what you are actually allowed to do with a model beyond trying it yourself.