How Linux schedules I/O, and why your server stalls under load

There's a particular kind of server problem that doesn't show up as high CPU, doesn't show up as low memory, and doesn't show up in a casual glance at top: everything just feels slow, intermittently, usually under load, and the box looks mostly idle the rest of the time. More often than people expect, the answer is sitting in the I/O scheduler, a part of the kernel most engineers never have to think about until the day they do.

This is what the I/O scheduler actually does, how it's changed, and how to tell when it's the thing making your server stall.

The symptom: everything's fine until it isn't

The classic presentation looks like this: a web server handling normal traffic fine, a background job kicks off — a backup, a log rotation, a batch import — and suddenly request latency on completely unrelated endpoints spikes for the duration of that job, even though CPU usage barely moves and there's plenty of free memory. The missing piece is disk I/O contention, and specifically, how fairly (or unfairly) the kernel is sharing access to the disk between the background job and everything else trying to read or write at the same time.

A short history of Linux I/O schedulers

For most of the 2.6 kernel era, Linux offered a handful of schedulers — CFQ (Completely Fair Queuing), deadline, and noop — built around a single request queue per disk, which made sense when "disk" meant a spinning platter with one read/write head and real seek-time costs to minimize.

NVMe changed the economics entirely: a modern SSD can handle tens of thousands of requests in parallel across multiple hardware queues, and a single software queue in front of it became a bottleneck rather than a help. The kernel's blk-mq (multi-queue block layer), which has been the default path for years now, replaced that single-queue model and brought a new generation of schedulers built for it.

The schedulers you'll actually meet: none, mq-deadline, bfq, kyber

Check what your system is running with:

cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none

• none — no reordering at all, requests go straight to the device. Correct choice for fast NVMe drives where the device's own internal queuing does a better job than software ever could.
• mq-deadline — the sensible default for most SATA SSDs and spinning disks. Guarantees a maximum wait time for any given request so nothing starves, without the complexity of trying to model fairness between processes.
• bfq (Budget Fair Queueing) — actively tries to give each process or cgroup a fair share of disk time, which is exactly what you want on a desktop (so a big file copy doesn't freeze your UI) and sometimes exactly what you want on a server with multiple tenants — at the cost of more CPU overhead per I/O operation.
• kyber — tries to hit target latencies for reads and writes separately, tuned for fast SSDs under mixed workloads; less commonly the default, but worth testing if mq-deadline isn't cutting it.

Reading iostat while the stall happens

iostat -x 1 is the first tool to reach for, watching it live during the next stall rather than after the fact:

Device  r/s   w/s   rkB/s   wkB/s   await  aqu-sz  %util
nvme0n1  42   980   3100  124000   38.4    12.1    99.8

The two numbers that matter most here are await (average time, in milliseconds, requests spend waiting plus being serviced) and %util pinned near 100. A healthy NVMe drive under light load shows await in the low single digits; seeing it climb into the tens of milliseconds while %util sits at 99–100% means requests are queuing up faster than the device (or the scheduler in front of it) can drain them — which is precisely the condition that turns an unrelated backup job into user-facing latency.

cgroups v2 and io.weight

If multiple services share one disk — common on a single VPS running several containers — cgroups v2's io controller lets you assign relative weights so one process can't monopolize disk bandwidth during a burst:

echo "250" > /sys/fs/cgroup/mygroup/io.weight

This only has teeth with a scheduler that respects cgroup weighting — bfq does, mq-deadline's cgroup support is more limited, and none bypasses the concept entirely since it doesn't reorder anything in software. If you're fighting a noisy-neighbor problem between containers on the same disk, that's the first thing to check: the weighting you configured might simply not be enforced by the scheduler you're running.

What to actually change on your server

For most servers on modern NVMe storage, none is already the right answer and the problem lies elsewhere — usually too many concurrent heavy jobs scheduled at once rather than the scheduler itself. For servers on SATA SSDs or spinning disks seeing exactly the "background job stalls everything else" pattern, switching from none or mq-deadline to bfq is the most common fix, trading a small amount of CPU overhead for much better fairness between competing I/O streams.

When the bottleneck isn't I/O at all

Before concluding it's the scheduler, rule out the more common impostors: a database missing an index that's doing a full table scan under load, a memory shortage causing swap activity that looks exactly like a disk stall because it is one, or simply a disk that's physically failing and throwing retries that iostat will also show as elevated await. The scheduler is a real and under-discussed piece of this puzzle, but it's the last thing to check, not the first — reach for iostat, confirm the pattern matches genuine queuing rather than one of those other causes, and only then start thinking about which scheduler is in front of the disk.