When a language model begins responding, the words appear one after another. Behind that experience, a serving system may be coordinating many simultaneous requests on a small number of accelerators. Some requests involve short questions; others carry long inputs or generate long replies. The server must choose which requests can run together, when to admit more work and how to avoid exhausting memory.
Those are scheduling decisions. They happen around the model rather than in its language generation, but they can have an immediate effect on how quickly and reliably a service responds.
Two phases with different needs
Many transformer inference systems distinguish an initial prefill phase, which processes the input context, from a decode phase that produces successive output tokens. The phases place different demands on computation, memory and scheduling. A long input can create substantial initial work; a long output can occupy generation resources for an extended period. Treating every request as an identical fixed-duration task is a poor approximation for many workloads.
The preferred policy depends on the service contract, the distribution of workloads and the cost of waiting.
Batching several requests can improve accelerator utilization. Yet waiting to assemble a large batch can delay the first request, and running too many long requests at once can hurt latency for everyone else. A useful scheduler balances throughput, the volume of work completed per unit time, with latency, how long an individual user waits.
Memory changes as the request progresses
Many serving systems store attention-related information in a key-value or KV cache. The memory consumed by active sequences can change as they generate tokens, so admitting one extra request is not always a simple matter of filling the next empty slot. When the memory limit is approached, the scheduler may need to delay, pause or manage requests differently. The available strategies depend on the model and serving architecture.
The interesting lesson is that an AI service depends on algorithms the language model never explicitly writes.
Suppose a GPU has room for several modest requests but one unusually long generation arrives. Admitting that request immediately might improve total work completed, while delaying it could keep short interactive requests responsive. The preferred policy depends on the service contract, the distribution of workloads and the cost of waiting. This is an optimization problem whose objective must reflect the experience or throughput the system is actually expected to deliver.
The algorithm has to fit the time scale
Some decisions must be made so quickly that a substantial mathematical search inside the request path would be counterproductive. Fast heuristics can be appropriate. Other decisions, such as configuring serving policies or placing batches of background jobs, may permit longer optimization runs. Learning-based policies and optimization-assisted tuning can also be investigated, provided their decision overhead is part of the evaluation.
The interesting lesson is that an AI service depends on algorithms the language model never explicitly writes. These algorithms manage shared computing resources under constraints, just as a factory scheduler manages machines and jobs. Data center workload scheduling expands the infrastructure view, and Custom Optimization is relevant to integrated optimization and problems unique to your organization.