A seven-billion-parameter AI model stored at 16-bit precision needs roughly 14GB just for its weights. Shrink those weights to four bits and the theoretical figure falls to about 3.5GB.
Nothing magical happened. The model did not forget three quarters of its vocabulary or sell several billion neurons on eBay.
It was quantised.
Quantisation is one of the technologies making powerful AI models practical on consumer GPUs, laptops, phones and edge devices. It also explains those mysterious labels such as FP16, INT8, Q4 and Q8 that appear whenever people discuss running an LLM locally.
AI models are enormous collections of numbers
A neural network contains parameters — values adjusted during training that collectively determine how it behaves.
Those numbers must be stored at some numerical precision.
A 16-bit representation uses 16 bits per parameter. Eight-bit uses eight. Four-bit uses four.
For a simplified 7-billion-parameter model:
16-bit: roughly 14GB
8-bit: roughly 7GB
4-bit: roughly 3.5GB
Actual memory requirements are higher because running a model also requires memory for things such as activations, buffers and the KV cache used when processing context.
But the principle is clear: fewer bits can dramatically reduce memory requirements.
Hugging Face's bitsandbytes documentation says eight-bit quantisation can approximately halve model-weight memory requirements, while four-bit methods compress them further.
Surely throwing away precision makes the AI worse?
Potentially.
Imagine recording everyone's height only to the nearest metre. Storage becomes wonderfully simple, but basketball scouting deteriorates rather rapidly.
Quantisation has the same fundamental problem: reducing precision means rounding values, and sufficiently crude rounding destroys useful information.
Modern quantisation techniques are cleverer than simply chopping digits off every parameter equally. Some retain greater precision for unusually important or sensitive values. Hugging Face's description of LLM.int8(), for example, explains that sensitive computations can remain at higher precision while much of the model is processed using eight-bit values.
Four-bit approaches go further.
The result can be a model dramatically smaller than its original version while retaining much of its practical capability.
Not all quantised models are equally good, though. The method used, calibration, model architecture, hardware and task all influence the trade-off.
Why this matters beyond hobbyists
Quantisation changes the economics of AI.
Smaller models can require less expensive GPU memory. More models can fit on the same server. Devices with limited memory can run models that previously required datacentre hardware.
That makes local AI considerably more interesting.
A laptop running a compact quantised model can potentially process private documents without continuously sending them to a cloud service. Robots, vehicles and industrial systems can perform AI inference even with poor connectivity. Smartphones can deliver features with lower latency.
The model has moved closer to the data.
Hugging Face now documents quantisation support across GPU and CPU environments, while techniques such as four-bit quantisation are widely incorporated into today's open-model ecosystem.
Smaller does not automatically mean faster
Here comes the annoying part.
Quantisation reduces memory, but it does not guarantee proportional speed improvements.
Hardware must efficiently support the numerical format. Some quantisation methods require additional operations to unpack or convert data. Hugging Face, for instance, notes circumstances in which bitsandbytes inference may be slower than other approaches or full-precision implementations.
Quality can also deteriorate, particularly as compression becomes more aggressive or on tasks sensitive to numerical precision.
So when somebody says a 30-billion-parameter model “runs on my laptop,” the interesting follow-up questions are: at what quantisation, at what speed, with what context length and with what quality loss?
Specifications have a habit of becoming much more educational after the fourth question.
Quantisation is nevertheless one of AI's most important enabling technologies because the future cannot consist entirely of buying larger GPUs.
Sometimes progress means building a bigger model.
Sometimes it means discovering how much of that bigness you never actually needed.