Setting up a workstation or home server to run AI locally

Learn in your browser · Chapter 01 of 07

Before any purchase, it helps to know which single specification actually predicts whether AI runs well on a machine, and it is not the one most people check first.

Two computers with the same processor speed can behave completely differently when you try to run AI locally, because the number that predicts success is not CPU speed. It is VRAM: the memory built into your graphics card, or the memory pool your GPU can reach. A language model, an image generator or a video generator all load their weights into that memory before they can do any work, and if the model does not fit, nothing else about the machine matters.

There are three memory pools you will meet, and they are not interchangeable. VRAM sits on a dedicated graphics card: fast, and usually the smallest pool in the system, from a few gigabytes on an older laptop to well over twenty gigabytes on a high-end card. System RAM is larger and shared with everything else the computer is doing; running a model from system RAM on the CPU alone works but is generally much slower than a GPU. Unified memory, used by Apple Silicon Macs, is a single pool shared between the CPU and GPU, so a Mac with a large amount of it can sometimes hold a bigger model than a laptop with a similarly sized but separate VRAM pool, because the GPU can reach the whole pool.

This workbook covers two different jobs, and they need very different amounts of that memory. Running a local text assistant, the kind covered in Running open-weight AI models on your own computer, for free, is comfortable on a modest dedicated GPU, sometimes even on a laptop's built-in graphics for the smallest models. Generating AI video locally is a heavier job, covered properly in Chapter 5, and it is worth deciding early which of these you actually need, because it changes almost every choice that follows.

nvidia-smi --query-gpu=name,memory.total --format=csv

Learn in your browser · Chapter 02 of 07

The same GPU can go into three very different kinds of machine, and the right shape depends on how and where you will actually use it, not just the specification sheet.

A laptop with a dedicated GPU is the only option if you genuinely need to carry the machine with you, but it pays a real price for that portability: laptop GPUs of the same name as a desktop card usually run at lower power and therefore lower performance, batteries do not like sustained heavy loads, and cooling in a thin case is a genuine limit on how hard you can push it for long. A desktop workstation trades portability for headroom: more room for a bigger GPU, better cooling, easier upgrades later, and no battery to protect. A home server, a machine left running permanently, usually headless, without its own screen, and reached over your network, is a different answer again: not primarily for interactive use, but for a model, an agent or a household AI tool that is always available, running around the clock.

Scroll sideways to see every column.

Consider two different people. One works from a cafe some days and wants to draft with a local model wherever they are: a laptop with a decent dedicated GPU is the honest answer, even knowing it will run some jobs more slowly than a desktop would. The other wants a model or agent available to every device in the house at any hour, without a laptop staying open on a desk: a small always-on home server suits that better, even at lower single-job performance.

Learn in your browser · Chapter 03 of 07

Once you know the shape of machine you want, the GPU inside it is the single highest-leverage choice you will make, and VRAM is still the first number to look at.

Beyond VRAM, three things genuinely matter. Architecture and generation: a newer generation of the same brand's GPUs is usually more efficient per watt and per gigabyte, so a recent mid-range card can outperform an older high-end one. Software ecosystem: NVIDIA's CUDA platform has, for years, had the deepest and most consistent support across AI tools, which matters more than raw specification, since a card only helps if the software you want to run actually supports it well. AMD's ROCm and Apple's Metal and MLX have both improved, but coverage across AI tools is patchier and worth checking against the specific tool you plan to use. New versus used: a previous-generation card with plenty of VRAM, bought used, is often better value than a newer card with less, because running out of memory is a hard stop, not a graceful slowdown.

Scroll sideways to see every column.

Two real shapes of GPU show the trade-off. A recent mid-range card with around 12 to 16 gigabytes of VRAM comfortably runs the small and medium local text models covered in Running open-weight AI models on your own computer, for free. An older but higher-VRAM card, or a Mac with a generously sized unified memory pool, can sometimes run a larger model than a newer card with less memory, simply because it has more room, even if each request runs more slowly. Capacity and raw speed are different numbers, and for whether something runs at all, capacity usually matters more.

Learn in your browser · Chapter 04 of 07

A powerful GPU is only as good as the power, cooling and case around it, and this is where people who priced the GPU carefully still get caught out.

A PSU, power supply unit, has a maximum wattage it can deliver, and every GPU manufacturer publishes a recommended minimum system PSU wattage for that specific card on its own product page. Treat that published number as a floor, not a target: add the rest of your system's real draw, the CPU under load, drives, fans, and leave genuine headroom above the total rather than buying to the exact minimum. A PSU running constantly near its limit runs hotter, less efficiently and less comfortably than one with room to spare.

Cooling and the room the machine lives in matter more for AI work than ordinary use, because generation jobs can run a GPU flat out for minutes or hours, not short bursts. Check that your case actually fits the GPU's physical length before buying either one, that air can genuinely move through the case rather than just past it, and that the room has some ventilation: sustained heavy use puts out real, noticeable heat, worth planning for rather than discovering afterwards.

A GPU working hard draws meaningfully more power than one sitting idle, for as long as the job runs. That is a genuine ongoing cost alongside the purchase price, scaling with how much you actually use the machine, worth factoring into the budget tiers in Chapter 6 rather than only the price on the box.

Scroll sideways to see every column.

Learn in your browser · Chapter 05 of 07

Local AI video generation is not just a bigger version of running a text model. It is a different, heavier problem, and the earlier chapters' numbers need revising upward before you buy anything with video in mind.

A text model processes one stream of tokens. A video generation model has to hold and process many frames at once, each a detailed image, across time as well as space, which is why open video-generation tools are typically far more demanding of VRAM than a text model of a similar-sounding size. Doubling a clip's length or resolution does not add a small amount of extra memory; it can push a job from comfortable to impossible on the same card.

Local video-generation tooling, as of this workbook's review, is overwhelmingly built and tested on NVIDIA's CUDA platform first. If AMD or Apple support for your chosen tool exists at all, check its own current documentation before buying hardware around it specifically, since this is one of the fastest-moving details in local AI and worth verifying fresh rather than assuming it matches the text-model situation in Chapter 2.

Once VRAM needs move into the tens of gigabytes, two honest routes open up. A single workstation-class or professional-tier GPU offers a large amount of VRAM in one card, at a real premium over consumer pricing. Running two or more consumer GPUs together, pooling their VRAM for a supported workload, is the other route some hobbyists take, and it asks more of your PSU, case, cooling and the software's own multi-GPU support than a single-card build does. Neither route is wrong; it depends on budget, build experience and patience with the extra setup work.

Learn in your browser · Chapter 06 of 07

Put the last four chapters together and a real purchase decision, not just a set of specifications, should now be within reach.

Three honest tiers cover most people, without pinning this workbook to prices that will date almost immediately. An entry tier suits everyday local text and image use: a modest, often previous-generation GPU in the lower VRAM range from Chapter 2's table, in whatever shape suits how you actually work. A mid tier suits someone who wants headroom for larger local models or occasional light video generation: a recent mid-range GPU with more VRAM, usually in a desktop for the cooling and upgrade room. A serious tier suits committed video-generation work or several demanding tools at once: a high-VRAM flagship, a workstation-class card, or more than one GPU together, almost always a desktop or dedicated server rather than a laptop.

If Chapter 1 pointed you towards a home server rather than a desktop or laptop, self-hosting platforms such as Proxmox, TrueNAS or Unraid let you run a model, an agent or several separate services on one always-on machine, with the GPU passed through to whichever one needs it. This is genuinely useful, general-purpose infrastructure, not something built specifically for AI, and it is worth exploring on its own terms rather than assuming it is required just because you have chosen the server shape.

Scroll sideways to see every column.

Learn in your browser · Chapter 07 of 07

A real hardware decision, not just a list of specifications, matched to how you actually plan to use it.

You can now explain why VRAM, not raw processor speed, decides what runs on a given machine, choose honestly between a laptop, a desktop workstation and a home server for your own situation, size a GPU against real VRAM tiers for both everyday local AI and the heavier case of AI video generation, and plan the PSU, cooling and case choices that actually make a GPU purchase work in practice.

https://trust-agent.ai/exam/workstation-or-home-server-for-ai

This is free educational content published by Mickai LTD, not professional or technical advice for your specific purchase, and Mickai accepts no liability for how you use it. Always check a component's own current specification sheet before buying.

Being able to say plainly what hardware your AI actually runs on, and who can reach it, is the same principle behind Mickai's approach to trustworthy AI: a human stays in control of the infrastructure, not just the model.

If you are reading this because your organisation is planning several of these machines rather than one, not just a single home workstation, Mickai builds bespoke AI workflows and code enhancements for businesses, running on Mickai's own sovereign operating system with zero data egress: your data and code never leave your own environment. See what that looks like at mickai.co.uk/bespoke.