What your machine can run

One question decides almost everything: does the model fit in memory? Everything else, speed and comfort, follows from it.

Where the model has to fit

• In the graphics card's memory (VRAM), if you have a dedicated one: this is the fastest case.
• In the unified memory of Apple-silicon Macs, shared between processor and graphics: a large part of it is usable by the model, which makes these machines very efficient for this use.
• In ordinary RAM, with the computation handled by the processor: it works, but noticeably more slowly.

The calculation rule

The course "Under the hood of LLMs" covered quantization: each parameter is stored on fewer bits. At 4 bits, the most common format for local use, count on roughly 0.6 GB per billion parameters, plus a margin for the conversation in progress. Orders of magnitude:

• A model of 7 to 8 billion parameters: about 5 GB. Fits on most 16 GB machines.
• Around 14 billion: about 9 GB. Comfortable with 16 GB of graphics memory, or 24 to 32 GB of unified memory.
• Around 30 billion: about 18 to 20 GB. You need a high-end machine.
• Around 70 billion: about 40 GB. Reserved for workstations and very well equipped Macs.

The conversation uses memory too

Every token in the conversation takes up memory on top of the model. A long pasted document can overwhelm a machine that ran the model without difficulty on a short question. The tools let you set the context length: a shorter one saves memory.

Speed

It is measured in tokens per second. Above about ten, reading is fluid; below three or four, the wait becomes painful. Speed depends mainly on memory bandwidth: for each token produced, the machine rereads all the active parameters. Hence the appeal of mixture-of-experts models, which activate only part of them, provided you have the memory to load the whole thing.

Key points

• Everything starts with memory: graphics, unified or RAM, in that order of speed.
• At 4 bits, about 0.6 GB per billion parameters, plus the conversation.
• Speed depends on memory bandwidth and the number of active parameters.