Fast CPU inference for EmbeddingGemma 2: 2.46x over PyTorch / ONNX Runtime (open source)
A short text query was stuck waiting 5.5 seconds behind one big image.
Now it waits 9 ms.
That's one of the things I fixed while making Google's EmbeddingGemma 2 fast on plain CPUs, with no GPU, for text, images, audio and video.
Starting from @QuixiAI's embeddinggemma.c, I rewrote the hot paths:
⚡ AVX-VNNI int8 matrix kernel, about 7x faster per core, on CPUs without AVX-512
🧠 Tiled online-softmax attention (GQA + sliding windows)
🧵 P/E-core aware scheduling, so small queries never starve behind big ones
💾 One copy of the weights in memory: ~1.4 GB vs ~4.8 GB for PyTorch
Results on a laptop i7-14700HX:
🏁 Faster than PyTorch and ONNX Runtime on 16/16 workloads
📈 2.46x geometric-mean speedup over the fastest alternative per test
🎯 Cosine to the official FP32 embeddings ≥ 0.9991
I've only measured it on one machine so far, so I'd love benchmark runs on your CPU 👇
https://github.com/Yazeedx0/embeddinggemma-cpu-max