Abstract
We present a systematic study of compiler-level optimizations on Google TPU v5e pods for serving large mixture-of-experts (MoE) models in bfloat16 precision. We achieve 3.4× latency reduction.
1. Introduction
Google's TPU v5e provides exceptional matrix multiply throughput. However, large MoE models suffer from expert dispatch communication bottlenecks. We demonstrate methods to overcome this.
2. Background
MoE expert dispatch involves all-to-all communication across chips. Standard JAX XLA compilation does not natively optimize for these communication patterns.
3. Methodology
Our methodology primarily involves micro-batch pipelining, custom XLA kernel fusion, and GSPMD sharding optimizations.
# Pipelined expert dispatch
pass
4. Results
We achieve sub-15ms TTFB (Time To First Byte) across a 64-batch inference load. End-to-end latency improved by 3.4×.
5. mX Compute
The mX compute layer orchestrates the prefill/decode phases independently across different TPU pod slices, leveraging different hardware profiles for compute-bound vs memory-bound tasks.
6. Conclusion
Targeted compiler optimizations for MoE routing on TPUs yield dramatic performance improvements, making 200B+ MoE models economically viable for production inference.