What does “Quantization” mean?
Converting model weights to lower precision formats such as 8-bit or 4-bit.
How Quantization Works
Weights stored as 16-bit numbers
Convert to 8-bit or 4-bit
Smaller file, less memory, faster.
Practical Example
A model file that is about 14 GB at 16-bit precision can shrink to roughly a quarter of that at 4-bit, so it can run on a laptop. Outputs may be slightly different from the original.
Why It Matters
Quantization makes it practical to run models on cheaper or local hardware. Testing quality on your own tasks is still important.
Common Questions About Quantization
Q: What do 4-bit and 8-bit mean?
How many bits are used to store each number. Fewer bits mean less memory and some loss of precision.
Q: Is it reversible?
Not exactly. Original precision cannot be perfectly recovered from the quantized version.
Q: Does it work for training too?
Mostly used for inference, though methods like QLoRA combine it with fine-tuning.