Exploring Mixture of Experts for sub-3B models targeting mobile deployment. How I converted Qwen3-0.6B to a sparse MoE with 4 experts, achieving comparable quality with a 19% parameter increase.

Deep dive into Cerebras WSE-3 wafer-scale architecture achieving 2000+ tokens per second. Compare with Baseten GPU optimization and Mercury diffusion models.