Identifying process causing lots of IOWait
Most common reason of high IOWait is bottleneck on disk io. Other reasons could be high network traffic or even terminal to output).
Once you see high IOWait problem comes to identifying the cause of this high IOWait.
To do that you can use block IO debugging capability of Linux kernel.
echo 1 > /proc/sys/vm/block_dump
dmesg | egrep "READ|WRITE|dirtied" | Â awk '{print $1}'| Â sort | uniq -c | sort -rn | head
1583 kjournald(2764): 545 kjournald(1023): 48 beam.smp(21021): 47 sendmail(20992): 28 crond(20974):
Now disable the block IO debugging
echo 0 > /proc/sys/vm/block_dump
Ignore kernel threads like kjournald (which is background thread for journaling all the writes happning throught ext3/4-fs). Ignore them and remaining user processes should be on your target list. (Here beam.smp and sendmail). Thanks Vijayr for pointing this out.Â
IOWait causing process will be sure in the output list. You can proceed from here with more investigations.
Sysstat-7.1.6 (Available in debian repos and not on RHEL/CentOS) 's pidstat gives per process IO statistics.
This gives snapshot of process doing IO every two seconds. This should help too.
iotop a top like utility which shows who is doing most io right now. This requires Linux 2.6.20 or more with TASK_DELAY_ACCT and TASK_IO_ACCOUNTING options enabled.
