The infrastructure behind inference.

My engineering focus is the boundary between inference runtimes and distributed platforms: deployment, routing, caching, performance and reliability.

01

Production LLM Inference

Building the serving layer between a model and the people using it.

  • Disaggregated prefill & decode
  • Request routing
  • Distributed serving
02

Kubernetes and Cloud Infrastructure

Making inference workloads easier to deploy, scale and operate.

  • Operators & lifecycle management
  • Workload orchestration
  • Reliability & observability
03

Inference Performance

Understanding where time and memory go, then improving the system.

  • KV-cache architecture
  • TTFT, ITL & goodput
  • Benchmarking & load testing

Current work: AWS SageMaker HyperPod Inference.

From systems to shared lessons.

Published architecture walkthroughs and technical deep dives.