Our monitoring plugin disk-io no longer measures iowait. What sounds like a step backwards is a step forward: iowait looks like a measure of overloaded storage, but it is relabelled idle time, and on any multi-core system it is, per the kernel developers themselves, simply useless. Triggered by a GitHub issue for disk-io on ZFS, we worked our way through the source of the kernel, sysstat, psutil, glances and OpenZFS - with a clear result.
We had iowait in disk-io as an additional signal, but we were never quite comfortable with what it told us in datacenter operations - the value was hard to pin down. The occasion to finally take a closer look came with Issue #1371: here disk-io reports values like iowait 377% on two freshly booted Proxmox servers in different datacenters. Both servers were practically idle, the disks had nothing to do at a few kilobytes per second, the VMs ran cleanly. What the two hosts had in common: ZFS. The reporter's correct assessment: "These high iowaits must be fantasy numbers."
The Linux kernel is unambiguous: iowait is not time a CPU spends on I/O, it is idle time under a different name. The kernel titles the relevant section in kernel/sched/core.c exactly that way and defines the value right below it:
IO-wait accounting, and how it's mostly bollocks (on SMP).
The idea behind IO-wait account is to account the idle time that we could
have spend running if it were not for IO. That is, if we were to improve the
storage performance, we'd have a proportional reduction in IO-wait time.iowait is therefore, by definition, idle time, just labelled differently. The code shows the same: the responsible routine is handed idle time and only decides which bucket it goes into. The kernel books a CPU's idle time to iowait instead of idle as soon as at least one task that last ran on this CPU is sleeping in an I/O wait state (kernel/sched/cputime.c):
void account_idle_time(u64 cputime)
{
u64 *cpustat = kcpustat_this_cpu->cpustat;
struct rq *rq = this_rq();
if (atomic_read(&rq->nr_iowait) > 0)
cpustat[CPUTIME_IOWAIT] += cputime;
else
cpustat[CPUTIME_IDLE] += cputime;
}The nr_iowait counter is incremented as soon as a task goes to sleep via io_schedule(), which means something entirely different from "the disk was busy". So iowait measures "compute time lost to waiting on I/O", not "how busy is the disk". The refreshingly honest comment in kernel/sched/core.c on the worth of this number on multi-core systems:
Worse, since the numbers are provided per CPU, they are sometimes
interpreted per CPU, and that is nonsensical. A blocked task isn't strictly
associated with any one particular CPU, it can wake to another CPU than it
blocked on. This means the per CPU IO-wait number is meaningless.iowait is therefore cleanly interpretable, if at all, only on a single-core system, and even there the value does not help in making statements about disks. Reason enough for us to remove the value from disk-io (again).
What about %util from iostat as a replacement? Here too it pays to look at the code. iostat computes %util from the io_ticks counter in /proc/diskstats (sysstat, rd_stats.c). Per the kernel (block/blk-core.c, update_io_ticks), io_ticks counts the wall-clock time during which at least one I/O request was in progress, not the queue depth.
For a classic single-queue disk like an HDD the value is usable, but for anything that processes requests in parallel (NVMe, SSDs, dm/md RAID and ZFS zvols) it is not: %util sits at 100% as soon as one request is continuously open, no matter how far the device is from its real limit. We keep %util as a pure trend metric (Linux, per device), but do not alert on it.
Why ZFS of all things drove disk-io to 377% is answered by the OpenZFS code: ZFS routes its synchronous I/O waits through cv_wait_io() and cv_timedwait_io() respectively, and these end up in the kernel at io_schedule() (module/os/linux/spl/spl-condvar.c). A thread waiting for a physical I/O (zio_wait()), or for the commit of a transaction group (txg_wait_synced()), thus sleeps over exactly the path that increments nr_iowait. OpenZFS even comments on this explicitly in module/zfs/txg.c:
Callers setting should_quiesce will use cv_wait_io() and be accounted for
as iowait time. Otherwise, the caller is understood to be idle and
cv_wait_sig() is used to prevent incorrectly inflating the system load average.Spread across many vCPUs, this pushes a disproportionate share of the otherwise empty idle time into the iowait statistic. An almost empty pool doing a few kilobytes per second thus ends up at several "saturated cores" while the disks are in fact doing nothing. The numbers are genuinely measured by the kernel. They just do not mean what one assumes.
glances doesFor comparison we pulled in the glances code, since like us it uses the psutil library. glances does not do it any better, which shows how widespread the assumption is: glances alerts on iowait by default, and against exactly the threshold most likely to cause false alarms on multi-core systems. The default threshold is 1 / number of CPU cores; above that the field is coloured, an event "High CPU I/O waiting" is written to the events list, and optionally a configured action is triggered (glances/config.py, glances/events_list.py).
iowait
We have therefore removed it from disk-io (in cpu-usage it no longer contributes to alerting either). disk-io measures the real throughput per device and compares it against the maximum ever observed, and it provides %util as a Linux trend metric with a clearly documented limitation. The old parameters --iowait-warning and --iowait-critical are kept for backward compatibility, but are ignored.
Instead we now deliver the signal that actually answers the question "is my storage slow": await, the average I/O latency per device. Unlike iowait, this is a physical quantity, robust against device parallelism, and exactly the number an admin needs when it matters.
await comes from the same fields in /proc/diskstats that iostat evaluates: the per-request service time summed up, including wait time in the queue, divided by the number of completed requests (sysstat, rd_stats.c). The result is the mean time in milliseconds an I/O takes from being issued to acknowledgement, split by read and write if needed (r_await and w_await).
await is an average ("average wait") and therefore hides outliers. A disk can show a harmless average while individual accesses hang for seconds. The number alone therefore does not tell you whether high latency comes from load or from a dying disk. We therefore read await together with the throughput per device, not as an isolated single value. But unlike iowait, await at least measures what it says on the tin. :-)
