Huawei releases MACaQ, its internal quantizer for local inference

MACaQ is Huawei’s model-agnostic quantization toolkit for producing compact, lower-precision models for local inference and serving. It supports text, image, video, and audio workloads across llama.cpp, vLLM, Diffusers, and PyTorch, with workflows for formats including 2-, 3-, and 4-bit GGUF and W4A4 weights and activations. The project reports that it can quantize a 122-billion-parameter model in 64 minutes using 32.5 GB of device memory. It also claims lower perplexity than MR-GPTQ and DuQuant++, or results matching their best Llama-3-8B perplexity with 11 times less quantization time.

The repository provides installation, calibration, quantization, and inference guides, with examples for dense and mixture-of-experts language models, diffusion-based image and video generation, and speech generation. GGUF models can run through the included llama.cpp workflow, including on CPUs; vLLM supports generation and HTTP serving, while Diffusers and PyTorch save compact checkpoints for their respective generation tasks. For Qwen3.8-27B in vLLM, the listed W4A4 result has perplexity of 2.8994 versus 2.8463 for BF16, with 2.33-times faster prefill and 2.58-times faster decoding. The project also notes that Ascend kernels are available through CANN-related integrations, and is distributed under the Apache 2.0 license.