Using iostat

Using iostat

iostat reports CPU utilization and disk I/O statistics. It is the primary tool for diagnosing disk-related performance problems. When your application is slow, CPU is mostly idle, and processes are in D state (uninterruptible sleep), iostat will show which disk is saturated. The extended output (-x) provides the metrics that matter: await (I/O latency), %util (disk saturation), and throughput.

iostat basics

sudo apt install -y sysstat

iostat             # One-shot summary since last reboot
iostat 1 5         # 1-second intervals, 5 samples (skip first line — averages since boot)

iostat basic output

avg-cpu:  %user  %system  %iowait  %steal   %idle
           34.5     8.2      2.1     0.0    55.2

Device    tps    kB_read/s  kB_wrtn/s  kB_read  kB_wrtn
sda      45.2      892.1      445.2   123456    67890
sdb       1.2        4.5     8924.1     5678  9876543

Extended statistics

# Extended stats with -x flag (the useful columns):
iostat -xz 1 3    # -x = extended, -z = skip idle devices

iostat -xz output with column meanings

Device  r/s   w/s  rkB/s   wkB/s  r_await  w_await  svctm  %util
sda    45.2  23.1  892.1   445.2     1.23     0.89   0.45   8.5%
sdb     1.2 892.4    4.5  8924.1   245.30    12.45  18.72  99.8%

Column meanings:
  r/s, w/s    = read/write operations per second (IOPS)
  rkB/s,wkB/s = read/write throughput in KB/s
  r_await     = average read latency in milliseconds
  w_await     = average write latency in milliseconds
  svctm       = service time (actual I/O time, no queue wait)
  %util       = percentage of time the device was busy

Identifying disk bottlenecks

# Interpreting iostat values:
# %util > 80%  = disk is likely a bottleneck
# %util 100%   = disk is saturated, all I/O is queuing

# Normal await times:
# SSD: <1ms for reads, <5ms for writes
# HDD: <20ms for sequential, <50ms acceptable

# await 245ms (like sdb above) = extremely overloaded HDD

# Find which processes are causing the I/O:
sudo iotop    # Interactive view of per-process I/O (requires installation)
sudo apt install -y iotop
sudo iotop -o    # Show only processes with active I/O

# Or use pidstat:
pidstat -d 1 5    # Per-process disk I/O every second

iotop output

TID    PRIO    USER    DISK READ    DISK WRITE    SWAPIN  IO>
1234   be/4  mysql    0.00 B/s   98.45 MB/s     0.00%  87.32%
5678   be/4  www-data  0.00 B/s    2.34 MB/s     0.00%   2.45%
→ mysql is responsible for 98MB/s of writes — check slow query log

Conclusion

The two numbers to watch in iostat -xz: %util above 80% indicates a disk bottleneck, and await above 20ms (HDD) or 5ms (SSD) indicates I/O latency that will slow applications. When you identify a saturated disk, use iotop to find which process is driving the I/O, then correlate with application logs to understand why. Common causes: database not using indexes (causes full table scans), log files growing unbounded, or a backup running during peak hours.