Inference engines explained

Learn in your browser · Chapter 01 of 07

Training builds a model. Inference uses it. Almost everything a builder does with AI is inference, and it has an engineering craft of its own.

Training adjusts a model's numbers, called weights or parameters, until it predicts text well. It happens on large clusters, once or in stages. Inference runs the finished model to produce output. The weights do not change while you use it, and the model remembers nothing between requests unless the earlier text is sent again.

An inference engine is the software that does this work. It loads the weights into fast memory, turns your text into tokens, runs the calculations, picks the next token and repeats. Around that loop it manages memory, schedules many users and offers an interface such as an HTTP API.

A model download is a bundle: the weights (by far the biggest part), a tokeniser that converts text to token numbers, and a configuration file describing the architecture, such as the layer count and context length. Weights come in a few common formats.

Scroll sideways to see every column.

Learn in your browser · Chapter 02 of 07

Every weight is a stored number. How many bits you spend on each one is your biggest lever over memory.

A parameter is one learned number. At 16-bit precision it takes 2 bytes, at 8-bit 1 byte and at 4-bit half a byte. Quantisation means storing weights with fewer bits than the model was trained with, so the file and its memory footprint shrink.

Scroll sideways to see every column.

The trade-off has three sides. Memory: smaller files fit on smaller devices. Speed: moving fewer bytes per token usually makes generation faster (chapter 4 explains why). Quality: rounding weights throws away some information. Lighter quantisation usually costs little and heavier costs more, but not equally for every task. The only reliable way to know is to test on your own work.

Real 4-bit formats also store small scaling values beside the weights, so the true size sits a little above the simple sum. Some engines can quantise the KV cache too (chapter 3). Some models are already published in a mixed low precision, so read the model card before you quantise anything twice.

Learn in your browser · Chapter 03 of 07

Whether a model runs comes down to one sum. Here it is in brief, from the engine's side.

Two things must fit in fast memory: the fixed weights, and the KV cache, which stores the keys and values attention has computed for every earlier token in each chat.

need = parameters x bytes_per_parameter        (weights)
     + cache_per_token x tokens x chats         (KV cache)
     + allowance                                (runtime overhead)

The per-token figure comes from the model's configuration. The allowance has no published size: like the workbook named below, this recap assumes the larger of 1 GB or 10% of weights plus cache, and keeps 10% of the device free as a safety margin. Measure, then replace both.

Illustrative, imaginary model, decimal GB: 8 billion parameters at 4-bit,
131,072 bytes of 16-bit cache per token, one 8,192-token chat

weights    8 billion x 0.5   = 4.00 GB
cache      131,072 x 8,192   = 1.07 GB
allowance  the 1 GB floor    = 1.00 GB
need                         = 6.07 GB, within 0.9 x 8 = 7.20 GB on an 8 GB card

Each extra chat adds its own cache while the weights stay put, so an engine's spare memory is a budget of cached tokens shared by all active chats (chapter 5). If it runs short, shorten the context, serve fewer chats, lower the precision, quantise the KV cache where supported, or offload layers to the CPU (chapter 6). Count every expert of a mixture-of-experts model, not only the active ones.

For the full method, its limits, and the memory that training and adapters need, see the workbook GPU memory maths for LLMs and SLMs.

Learn in your browser · Chapter 04 of 07

When someone says a model is fast, ask: fast at what? Generation has two very different phases.

Prefill processes your whole prompt in one pass. All the prompt tokens can be handled in parallel, so it is dominated by raw arithmetic and takes longer as prompts grow. It ends when the first output token appears. That wait is the time to first token.

Decode then writes the reply one token at a time. Each token depends on the one before, so the steps cannot run in parallel. Every step must read all the active weights, and the growing KV cache, from memory. That makes decode usually limited by memory bandwidth, the rate at which data moves, rather than by arithmetic. It is why replies stream out word by word.

A rule of thumb, as an upper bound: if each decode step must read W gigabytes and your memory delivers B gigabytes per second, one chat cannot exceed roughly B divided by W tokens per second. Halve W by quantising and the ceiling roughly doubles. Real speed is lower, so measure your own setup.

Scroll sideways to see every column.

Learn in your browser · Chapter 05 of 07

One user's request leaves most of a GPU idle. Serving engines exist to keep it busy without wasting memory.

Batching runs several requests through the model together. Because decode is limited by reading the weights, and one read can serve every sequence in the batch, adding sequences costs little extra time per step. That is how throughput rises.

Static batching waits for a batch to fill and runs it until the slowest request finishes, leaving finished slots idle. Continuous batching, also called iteration-level scheduling, re-forms the batch after every step: finished requests leave and waiting ones join at once. A 2022 systems paper described it, and it is a core feature of engines built for many users.

Paged attention attacks memory waste. Reserving one large contiguous block per request for its KV cache leaves gaps and duplicates. Paged attention splits the cache into small fixed-size blocks allocated as needed, borrowing the idea of virtual memory from operating systems. Less waste means more requests fit, which means bigger batches.

Speculative decoding speeds up decode. A small, fast draft (a separate small model, or a module built into the main one) guesses several tokens ahead. The large model checks them all in one parallel pass and accepts those it would have produced itself. Done correctly the output distribution is unchanged, so it alters speed, not answers. The gain depends on how often guesses are accepted.

Scroll sideways to see every column.

Learn in your browser · Chapter 06 of 07

The right hardware follows from the memory sum, not the other way round.

GPU memory (VRAM) is fast and limited. If weights and cache fit, you get the best speed. CPU with system RAM can hold much larger models, but it generally moves data more slowly, so decode is slower. Unified memory lets the CPU and GPU share one pool, so more of the machine's memory is usable by the model. Its speed depends on that pool's bandwidth, so check the specification. Hybrid offload puts some layers on the GPU and the rest on the CPU, so a model bigger than your VRAM can still run, more slowly.

Three well-known open-source engines make useful examples. Each fact below was checked against the project's own documentation on 25 September 2026. Engines change quickly, so check current documentation before relying on any of it.

Scroll sideways to see every column.

Scroll sideways to see every column.

Neither is better in general. Decide by data sensitivity, volume, and the model size you need.

Learn in your browser · Chapter 07 of 07

Most engines expose the model as a small web server. That is convenient, and it is also an attack surface.

The common pattern: the engine loads the model once and listens on an address and port. Your apps send JSON over HTTP and receive text back, often streamed. Many engines copy the chat-completions request format popularised by the big commercial chat APIs, so client code written for one can often be pointed at another by changing the base address.

# Illustrative request. The port and model name vary by engine.
curl 127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "local-model", "messages": [{"role": "user", "content": "Say hello in five words."}]}'

Bind to localhost. The address 127.0.0.1 means only programs on the same machine can connect. The address 0.0.0.0 means every network interface, which can expose your model to your whole network or the internet. The llama.cpp server documents a default of 127.0.0.1, and Ollama's FAQ says it binds 127.0.0.1 by default. Some container examples use 0.0.0.0 inside the container, so publish ports with care, and check the default of whatever you run.

1. Keep the server bound to localhost unless you have a reason not to.
2. If others need access, put an authenticating, encrypted reverse proxy in front. Do not simply open the port.
3. Turn on the engine's API key option, and read what it covers. vLLM's documentation warns that its key protects only some routes.
4. Do not enable experimental built-in tools, such as file access or shell commands, on a server others can reach. The llama.cpp server documentation warns against it in untrusted environments.
5. Decide what is logged. Prompts and replies can hold personal or confidential data. Log metadata (time, model, token counts, errors) by default, keep content logs off unless needed, and protect and expire any you keep.
6. Keep the engine and its libraries updated, and download weights only from the publisher's official page.

Next, see how reasoning models spend extra computation, and how to judge a new model the day it lands.