All notes

Why vLLM and SGLang feel like operating systems

First, what are they? vLLM and SGLang are open-source tools for serving large language models. When we open an AI product, ask a question and wait for an answer, thousands of other people may be doing the same thing at once.

The model generates answers. But deciding which request runs next, which requests can run together and what happens when GPU memory runs short takes another layer of coordination. That is part of what vLLM and SGLang do.

This is what makes the operating-system analogy useful. An operating system lets many programs share a computer’s resources; these inference engines let many user requests share the GPUs running a model. Requests queue up when the system is busy, run in batches when possible and reuse earlier computation where they can.

It is an analogy, of course. These engines do not replace the computer’s operating system. They are taking on more of the resource scheduling needed to serve large language models.