Check out my YouTube video of realtime latency graphs using DTrace and gnuplot.

Scripts here https://github.com/bahamas10/realtime-dtrace-visualization

The graph on the left shows read system call latency, and the graph on the right shows write system call latency (syscall::read*:entry and syscall::write*:entry respectively). The bottom portion is a modified oneliner to show files opened by a process (excluding gnuplot) taken from Brendan Gregg’s DTrace oneliners.

At 0:45, I induced latency by reading /usr/share/dict/words and writing the contents out to the screen in a 1 second loop. This becomes visible in large spikes on the graph showing write latency, but the read latency graph is almost unaffected.

The X-axis on both graphs show time (ticking every second) in the form of MM:SS. The Y-axis represents the time in milliseconds, and each point is a 1 second average aggregation of the latency.

Because of the nature of gnuplot in this experiment, the Y-axis adapts to the visible metrics, so when the latency is induced, the Y-axis grows from ~.005ms to ~2ms.