---
title: 'Why we stopped measuring iowait'
url: 'https://www.linuxfabrik.ch/en/blog/mystic-iowait'
markdown: 'https://www.linuxfabrik.ch/en/blog/mystic-iowait.md'
lang: en
date: '2026-07-13'
description: 'iowait looks like a measure of overloaded storage, but it is relabelled idle time and, per the kernel, useless on multi-core systems. Why we removed it from disk-io.'
taxonomy:
  category:
    - blog
  tag:
    - linux
    - monitoring-plugins
  archives_month:
    - jul_2026
  archives_year:
    - '2026'
---

# Why we stopped measuring iowait

[ Home ](https://www.linuxfabrik.ch/en)     [ Blog ](https://www.linuxfabrik.ch/en/blog)      Why we stopped measuring iowait    

Our monitoring plugin [disk-io](https://linuxfabrik.github.io/monitoring-plugins/check-plugins/disk-io/) no longer measures iowait. What sounds like a step backwards is a step forward: iowait looks like a measure of overloaded storage, but it is relabelled idle time, and on any multi-core system it is, per the kernel developers themselves, simply useless. Triggered by a GitHub issue for `disk-io` on ZFS, we worked our way through the source of the kernel, `sysstat`, `psutil`, `glances` and OpenZFS - with a clear result.

## The iowait mystery

We had iowait in `disk-io` as an additional signal, but we were never quite comfortable with what it told us in datacenter operations - the value was hard to pin down. The occasion to finally take a closer look came with [Issue #1371](https://github.com/Linuxfabrik/monitoring-plugins/issues/1371): here `disk-io` reports values like `iowait 377%` on two freshly booted Proxmox servers in different datacenters. Both servers were practically idle, the disks had nothing to do at a few kilobytes per second, the VMs ran cleanly. What the two hosts had in common: ZFS. The reporter's correct assessment: "These high iowaits must be fantasy numbers."

## What iowait really is

The Linux kernel is unambiguous: **iowait is not time a CPU spends on I/O, it is idle time under a different name.** The kernel titles the relevant section in `kernel/sched/core.c` exactly that way and defines the value right below it:

```text
IO-wait accounting, and how it's mostly bollocks (on SMP).

The idea behind IO-wait account is to account the idle time that we could
have spend running if it were not for IO. That is, if we were to improve the
storage performance, we'd have a proportional reduction in IO-wait time.
```

iowait is therefore, by definition, idle time, just labelled differently. The code shows the same: the responsible routine is handed idle time and only decides which bucket it goes into. The kernel books a CPU's idle time to iowait instead of idle as soon as at least one task that last ran on this CPU is sleeping in an I/O wait state (`kernel/sched/cputime.c`):

```c
void account_idle_time(u64 cputime)
{
    u64 *cpustat = kcpustat_this_cpu->cpustat;
    struct rq *rq = this_rq();

    if (atomic_read(&rq->nr_iowait) > 0)
        cpustat[CPUTIME_IOWAIT] += cputime;
    else
        cpustat[CPUTIME_IDLE] += cputime;
}
```

The `nr_iowait` counter is incremented as soon as a task goes to sleep via `io_schedule()`, which means something entirely different from "the disk was busy". So iowait measures "compute time lost to waiting on I/O", not "how busy is the disk". The refreshingly honest comment in `kernel/sched/core.c` on the worth of this number on multi-core systems:

```text
Worse, since the numbers are provided per CPU, they are sometimes
interpreted per CPU, and that is nonsensical. A blocked task isn't strictly
associated with any one particular CPU, it can wake to another CPU than it
blocked on. This means the per CPU IO-wait number is meaningless.
```

iowait is therefore cleanly interpretable, if at all, only on a single-core system, and even there the value does not help in making statements about disks. Reason enough for us to remove the value from `disk-io` (again).

## %util

What about `%util` from `iostat` as a replacement? Here too it pays to look at the code. `iostat` computes `%util` from the `io_ticks` counter in `/proc/diskstats` (`sysstat`, `rd_stats.c`). Per the kernel (`block/blk-core.c`, `update_io_ticks`), `io_ticks` counts the wall-clock time during which at least one I/O request was in progress, **not** the queue depth.

For a classic single-queue disk like an HDD the value is usable, but for anything that processes requests in parallel (NVMe, SSDs, `dm`/`md` RAID and ZFS zvols) it is not: `%util` sits at 100% as soon as one request is continuously open, no matter how far the device is from its real limit. We keep `%util` as a pure trend metric (Linux, per device), but do not alert on it.

## Why ZFS produces such absurd numbers

Why ZFS of all things drove `disk-io` to 377% is answered by the OpenZFS code: ZFS routes its synchronous I/O waits through `cv_wait_io()` and `cv_timedwait_io()` respectively, and these end up in the kernel at `io_schedule()` (`module/os/linux/spl/spl-condvar.c`). A thread waiting for a physical I/O (`zio_wait()`), or for the commit of a transaction group (`txg_wait_synced()`), thus sleeps over exactly the path that increments `nr_iowait`. OpenZFS even comments on this explicitly in `module/zfs/txg.c`:

```text
Callers setting should_quiesce will use cv_wait_io() and be accounted for
as iowait time. Otherwise, the caller is understood to be idle and
cv_wait_sig() is used to prevent incorrectly inflating the system load average.
```

Spread across many vCPUs, this pushes a disproportionate share of the otherwise empty idle time into the iowait statistic. An almost empty pool doing a few kilobytes per second thus ends up at several "saturated cores" while the disks are in fact doing nothing. The numbers are genuinely measured by the kernel. They just do not mean what one assumes.

## What `glances` does

For comparison we pulled in the `glances` code, since like us it uses the `psutil` library. `glances` does not do it any better, which shows how widespread the assumption is: `glances` alerts on iowait by default, and against exactly the threshold most likely to cause false alarms on multi-core systems. The default threshold is 1 / number of CPU cores; above that the field is coloured, an event "High CPU I/O waiting" is written to the events list, and optionally a configured action is triggered (`glances/config.py`, `glances/events_list.py`).

## Conclusion

iowait

- does not measure disk saturation, but relabelled idle time,
- is, per the kernel, unreliable on all multi-core systems,
- regularly produces fantasy values on ZFS,
- and is therefore useful neither as an alerting nor as a reliable trend metric.

We have therefore removed it from `disk-io` (in `cpu-usage` it no longer contributes to alerting either). `disk-io` measures the real throughput per device and compares it against the maximum ever observed, and it provides `%util` as a Linux trend metric with a clearly documented limitation. The old parameters `--iowait-warning` and `--iowait-critical` are kept for backward compatibility, but are ignored.

## What's next

Instead we now deliver the signal that actually answers the question "is my storage slow": `await`, the average I/O latency per device. Unlike iowait, this is a physical quantity, robust against device parallelism, and exactly the number an admin needs when it matters.

`await` comes from the same fields in `/proc/diskstats` that `iostat` evaluates: the per-request service time summed up, including wait time in the queue, divided by the number of completed requests (`sysstat`, `rd_stats.c`). The result is the mean time in milliseconds an I/O takes from being issued to acknowledgement, split by read and write if needed (`r_await` and `w_await`).

`await` is an average ("average wait") and therefore hides outliers. A disk can show a harmless average while individual accesses hang for seconds. The number alone therefore does not tell you whether high latency comes from load or from a dying disk. We therefore read `await` together with the throughput per device, not as an isolated single value. But unlike iowait, `await` at least measures what it says on the tin. :-)

 [ Previous Post](https://www.linuxfabrik.ch/en/blog/kernel-vulnerability-ssh-keysign-pwn)

#### Tags

  [ansible](https://www.linuxfabrik.ch/en/blog/tag:ansible) [checklistfabrik](https://www.linuxfabrik.ch/en/blog/tag:checklistfabrik) [cloud](https://www.linuxfabrik.ch/en/blog/tag:cloud) [datacenter](https://www.linuxfabrik.ch/en/blog/tag:datacenter) [event](https://www.linuxfabrik.ch/en/blog/tag:event) [firewallfabrik](https://www.linuxfabrik.ch/en/blog/tag:firewallfabrik) [graylog](https://www.linuxfabrik.ch/en/blog/tag:graylog) [icinga](https://www.linuxfabrik.ch/en/blog/tag:icinga) [lfops](https://www.linuxfabrik.ch/en/blog/tag:lfops) [linux](https://www.linuxfabrik.ch/en/blog/tag:linux) [monitoring-plugins](https://www.linuxfabrik.ch/en/blog/tag:monitoring-plugins) [news](https://www.linuxfabrik.ch/en/blog/tag:news) [nextcloud](https://www.linuxfabrik.ch/en/blog/tag:nextcloud) [security](https://www.linuxfabrik.ch/en/blog/tag:security) 

#### Archives

- [2026](https://www.linuxfabrik.ch/en/blog/archives_year:2026)
- [2025](https://www.linuxfabrik.ch/en/blog/archives_year:2025)
- [2024](https://www.linuxfabrik.ch/en/blog/archives_year:2024)
- [2023](https://www.linuxfabrik.ch/en/blog/archives_year:2023)
- [2022](https://www.linuxfabrik.ch/en/blog/archives_year:2022)
- [2021](https://www.linuxfabrik.ch/en/blog/archives_year:2021)
- [2020](https://www.linuxfabrik.ch/en/blog/archives_year:2020)

#### Syndicate

 [ Atom](https://www.linuxfabrik.ch/en/blog.atom) [ RSS](https://www.linuxfabrik.ch/en/blog.rss)

---

## Navigation

- Parent: [Blog](https://www.linuxfabrik.ch/en/blog.md)
- Next: [Kernel Vulnerability 'ssh-keysign-pwn' (CVE-2026-46333)](https://www.linuxfabrik.ch/en/blog/kernel-vulnerability-ssh-keysign-pwn.md)