What It Takes to Run a Model

This section explains which parts of a computer a language model needs to run.

Storage and Memory

A server is a computer, and it is built from the same main parts as the computers in a classroom. Those parts include a central processing unit, a graphics processing unit, memory and storage. Storage is the part that keeps files, and it holds them even after the computer is turned off. Memory is the part of a computer that holds the programs and data being used right now, and it is much faster than storage. Memory loses what it holds when the power goes off. Before a program can run, the computer copies the program and its data from storage into memory.

For example, a language model's parameter file sits in storage while nothing is happening, and the computer has to load it into memory before the model can predict a single token, and the parameter file stays loaded in memory while the model runs. A parameter file that is only saved on a drive is not in memory yet, because nothing has copied it there. A model needs enough memory to hold all of its parameters at once.

The CPU and the GPU

The CPU, or central processing unit, is the part of a computer whose main job is to run programs. It is built to finish one series of steps quickly, and it works on only a few tens of them at the same time.

A GPU, or graphics processing unit, is a part built to carry out thousands of simple calculations at the same time. For example, a model's prediction is made of very large groups of multiplications, and many of them can be done at the same time, which is the kind of work a GPU is built for. A CPU can do the same arithmetic, but it works on far fewer calculations at once, so it produces tokens much more slowly. A GPU is not the part that runs the rest of the computer, because the operating system and the other programs run on the CPU.

Model Size and Lab 1

In Lab 1, the three panes used three models from Mistral: Ministral 3B, Ministral 14B and Mistral Large 3. Ministral 3B has about 3.4 billion parameters for language, Ministral 14B has about 13.5 billion, and Mistral Large 3 has 675 billion in total. A bigger parameter file needs more memory to load. A model predicts one token at a time, and its parameters are read again for each new token. Mistral Large 3 uses about 41 billion of its parameters for each token, which is still more than either Ministral model has in total.

The seconds and tokens per second that students recorded in Lab 1 measured the whole trip through Mistral's service. Speed through a service also depends on the machines the provider uses and how it runs the model on them. For this reason, a slower pane fits with a bigger model, but the Lab 1 numbers alone do not show that size was the only cause.

Two Machines Side by Side

Figure 1 shows the inside of a server beside two spec cards. The first card describes a typical school Chromebook with 4 GB or 8 GB of memory and 32 GB or 64 GB of storage. Its GPU is built into the processor chip and shares the system memory with everything else on the computer. The second card describes an example machine that can run a model, with an NVIDIA GeForce RTX 4090 GPU that has 24 GB of its own memory and with 96 GB of system memory. This machine is an example and not the class server, because the class models run on Mistral's servers.

The model card for Ministral 14B says it fits in 24 GB of GPU memory, so it fits on the example machine's GPU and not on either Chromebook. Ministral 3B, the smallest class model, fits in 8 GB of GPU memory in the form Mistral publishes, which is all the memory the larger Chromebook has. The same model card says that a more compressed version needs less memory. A very small model could therefore load on a Chromebook, and the Chromebook would produce its tokens slowly. Mistral Large 3 needs far more, and its model card names a machine with eight NVIDIA H200 GPUs, each with 141 GB of memory.

Summary

A computer has to load a model from storage into memory before the model can run, and the memory must be large enough to hold all of its parameters. A GPU carries out the many simple calculations of prediction at the same time, while the CPU runs the rest of the computer and can run a model only slowly. A school Chromebook has enough memory for a very small model at most, and the class models need far more.

Sources: CompTIA, Tech+ FC0-U71 Exam Objectives (2023); Franchitti, Introduction to Computer Science, OpenStax (2024); NVIDIA, CUDA Programming Guide (2026); Verma and Vaidya, "Mastering LLM Techniques: Inference Optimization," NVIDIA (2023); NVIDIA, GeForce RTX 4090 and H200 specification pages; Lenovo, 100e Chromebook Gen 4 PSREF (2023); Mistral AI, model cards and "Introducing Mistral 3" (2025); MuggsOfCompSci, Lab 1 records and build spec (2026).