mirror of
https://github.com/wassname/vllm.git
synced 2026-06-27 18:27:02 +08:00
Qubitium-ModelCloud
cd1d3c3df8
Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com>
300 B
300 B
(quantization-index)=
Quantization
Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
:::{toctree} :caption: Contents :maxdepth: 1
supported_hardware auto_awq bnb gguf gptqmodel int4 int8 fp8 quantized_kvcache :::