Production LLM Inference
Building the serving layer between a model and the people using it.
- Disaggregated prefill & decode
- Request routing
- Distributed serving
Work
My engineering focus is the boundary between inference runtimes and distributed platforms: deployment, routing, caching, performance and reliability.
Building the serving layer between a model and the people using it.
Making inference workloads easier to deploy, scale and operate.
Understanding where time and memory go, then improving the system.
Current work: AWS SageMaker HyperPod Inference.
Publications
Published architecture walkthroughs and technical deep dives.
How managed NVIDIA Dynamo on HyperPod supports multi-node serving, disaggregated prefill and decode, and KV-aware routing, with benchmarks on Qwen models.
Read articleA practical look at the KV-cache transfer path, routing and benchmarks for disaggregated inference with vLLM, LMCache, NIXL and EFA.
Read articleAn architecture and deployment walkthrough for running prefill and decode on separate GPU pools, with KV-cache transfer over EFA RDMA.
Read articleSimplifying the path from a Kubernetes cluster to model serving through managed operator installation, lifecycle management and deployment options.
Read article