Transformers documentation
MXFP4
MXFP4
MXFP4 is the 4-bit floating-point format from the Open Compute Project Microscaling (MX) specification. It stores E2M1 values in blocks that share one power-of-two (E8M0) scale. In Transformers, MXFP4 quantizes the mixture-of-experts layers of GPT-OSS models, so use it to run GPT-OSS with less memory.
The table below summarizes what MXFP4 supports in Transformers.
| Config | Mxfp4Config |
quant_method | mxfp4 |
| Workflow | Quantize while loading, load pre-quantized |
| Bits | 4 |
| Hardware | CUDA (compute capability 7.5+), XPU, CPU |
| Save | ✅ |
| Fine-tune | ❌ |
| Auto-compile | ❌ |
Install MXFP4
Transformers downloads the MXFP4 Triton kernels from the kernels-community/gpt-oss-triton-kernels repository on the Hub. Install the kernels library, Accelerate, and Triton. CUDA needs Triton 3.4.0 or later, and XPU and CPU need Triton 3.5.0 or later.
pip install -U "transformers[kernels]" accelerate "triton>=3.4.0"Quantize a model with MXFP4
Create a Mxfp4Config and pass it to from_pretrained() to quantize a bf16 GPT-OSS checkpoint while it loads. MXFP4 quantizes only the expert weights. The attention layers, router, embeddings, and lm_head keep their original precision.
from transformers import AutoModelForCausalLM, Mxfp4Config
quantization_config = Mxfp4Config()
model = AutoModelForCausalLM.from_pretrained(
"unsloth/gpt-oss-20b-BF16",
quantization_config=quantization_config,
device_map="cuda",
)If the hardware or packages MXFP4 needs aren’t available, set dequantize=True to load the model in bf16 instead.
Load a pre-quantized checkpoint
The openai/gpt-oss-20b and openai/gpt-oss-120b checkpoints are stored in MXFP4, and their config sets quant_method to mxfp4. Load them without a Mxfp4Config.
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cuda")If your hardware or packages can’t run MXFP4, Transformers logs a warning and dequantizes a pre-quantized checkpoint to bf16 instead of raising an error.
Choose a kernel backend
By default, MXFP4 runs the expert layers with the Triton kernels on CUDA and XPU. Pass use_kernels=True to from_pretrained() to run them with the MegaBlocks kernel from the Hub instead.
model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", device_map="cpu", use_kernels=True)The backend you get depends on the device.
| Device | use_kernels=False | use_kernels=True |
|---|---|---|
| CUDA | MXFP4 with Triton kernels | Dequantizes to bf16 |
| XPU | MXFP4 with Triton kernels | MXFP4 with MegaBlocks |
| CPU | Dequantizes to bf16 | MXFP4 with MegaBlocks |
Limitations
- MXFP4 quantizes only GPT-OSS expert layers. Other models load without any quantized layers, and Transformers logs a warning.
- MXFP4 needs a CUDA GPU with compute capability 7.5 or later, an XPU, or a CPU. Quantizing a model on unsupported hardware or without Triton or kernels raises a
ValueError. Quantizing a model on a machine with another accelerator, like MPS, raises aRuntimeError. - Quantizing a model while it loads can’t offload to disk. A
device_mapthat includes"disk"raises aValueError. - MXFP4 models can’t be fine-tuned. Load the model with
dequantize=Trueto train it in bf16.
Resources
- OCP Microscaling Formats (MX) v1.0 specification
- MXFP4 quantization section of the gpt-oss tricks blog post
- kernels-community/gpt-oss-triton-kernels Hub kernel