kernel-builder and kernels libraries; so you can focus on optimizing performance rather than spending time on setup.kernels.get_kernel() 
From the available metrics I currently have, I looked at how much quantizing only the mlp.down_proj layers impacts end-to-end perplexity compared to the original float model and to quantizing all linear layers.
I used asymmetric W4A16 quantization and evaluated on the same held-out dataset used for the sensitivity profiling.
Here’s a summary of the results:
| Model | Original PPL | mlp.down_proj quantized PPL |
All linear layers quantized PPL (except lm_head) |
|---|---|---|---|
| Gemma-2B | 16.80 | 58.29 | 76.69 |
| LLaMA-3.2-3B | 13.73 | 15.11 | 17.32 |
| Qwen-3-4B | 20.10 | 22.73 | 29.70 |
Observations:
mlp.down_proj layers already causes a large perplexity increase (from 16.80 -> 58.29), showing they’re a major bottleneck for precision loss.mlp.down_proj, so the relative contribution is less pronounced.It would also be interesting to run evaluations of these models on downstream benchmarks to see how these perplexity changes translate into real-task performance differences.
For a detailed breakdown of the divergence-based method used to measure layer sensitivity, you can read my blog post here: https://huggingface.co/blog/badaoui/sensitivity-aware-mixed-precision-quantizer-v1
