Transformers documentation

MXFP4

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

MXFP4

MXFP4 is the 4-bit floating-point format from the Open Compute Project Microscaling (MX) specification. It stores E2M1 values in blocks that share one power-of-two (E8M0) scale. In Transformers, MXFP4 quantizes the mixture-of-experts layers of GPT-OSS models, so use it to run GPT-OSS with less memory.

The table below summarizes what MXFP4 supports in Transformers.

ConfigMxfp4Config
quant_methodmxfp4
WorkflowQuantize while loading, load pre-quantized
Bits4
HardwareCUDA (compute capability 7.5+), XPU, CPU
Save✅
Fine-tune❌
Auto-compile❌

Install MXFP4

Transformers downloads the MXFP4 Triton kernels from the kernels-community/gpt-oss-triton-kernels repository on the Hub. Install the kernels library, Accelerate, and Triton. CUDA needs Triton 3.4.0 or later, and XPU and CPU need Triton 3.5.0 or later.

pip install -U "transformers[kernels]" accelerate "triton>=3.4.0"

Quantize a model with MXFP4

Create a Mxfp4Config and pass it to from_pretrained() to quantize a bf16 GPT-OSS checkpoint while it loads. MXFP4 quantizes only the expert weights. The attention layers, router, embeddings, and lm_head keep their original precision.

from transformers import AutoModelForCausalLM, Mxfp4Config

quantization_config = Mxfp4Config()
model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gpt-oss-20b-BF16",
    quantization_config=quantization_config,
    device_map="cuda",
)

If the hardware or packages MXFP4 needs aren’t available, set dequantize=True to load the model in bf16 instead.

Load a pre-quantized checkpoint

The openai/gpt-oss-20b and openai/gpt-oss-120b checkpoints are stored in MXFP4, and their config sets quant_method to mxfp4. Load them without a Mxfp4Config.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cuda")

If your hardware or packages can’t run MXFP4, Transformers logs a warning and dequantizes a pre-quantized checkpoint to bf16 instead of raising an error.

Choose a kernel backend

By default, MXFP4 runs the expert layers with the Triton kernels on CUDA and XPU. Pass use_kernels=True to from_pretrained() to run them with the MegaBlocks kernel from the Hub instead.

model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cpu", use_kernels=True)

The backend you get depends on the device.

Deviceuse_kernels=Falseuse_kernels=True
CUDAMXFP4 with Triton kernelsDequantizes to bf16
XPUMXFP4 with Triton kernelsMXFP4 with MegaBlocks
CPUDequantizes to bf16MXFP4 with MegaBlocks

Limitations

  • MXFP4 quantizes only GPT-OSS expert layers. Other models load without any quantized layers, and Transformers logs a warning.
  • MXFP4 needs a CUDA GPU with compute capability 7.5 or later, an XPU, or a CPU. Quantizing a model on unsupported hardware or without Triton or kernels raises a ValueError. Quantizing a model on a machine with another accelerator, like MPS, raises a RuntimeError.
  • Quantizing a model while it loads can’t offload to disk. A device_map that includes "disk" raises a ValueError.
  • MXFP4 models can’t be fine-tuned. Load the model with dequantize=True to train it in bf16.

Resources

Update on GitHub