Check out my YouTube video of realtime latency graphs using DTrace and gnuplot.

Scripts here https://github.com/bahamas10/realtime-dtrace-visualization

The graph on the left shows read system call latency, and the graph on the right shows write system call latency (syscall::read*:entry and syscall::write*:entry respectively). The bottom portion is a modified oneliner to show files opened by a process (excluding gnuplot) taken from Brendan Gregg’s DTrace oneliners.

At 0:45, I induced latency by reading /usr/share/dict/words and writing the contents out to the screen in a 1 second loop. This becomes visible in large spikes on the graph showing write latency, but the read latency graph is almost unaffected.

The X-axis on both graphs show time (ticking every second) in the form of MM:SS. The Y-axis represents the time in milliseconds, and each point is a 1 second average aggregation of the latency.

Because of the nature of gnuplot in this experiment, the Y-axis adapts to the visible metrics, so when the latency is induced, the Y-axis grows from ~.005ms to ~2ms.

Working on performance analysis with an interpreted language poses some interesting problems. Fortunately, Node.js has built in support for DTrace thanks to the guys at Joyent. These built-in facilities make it easy to trace Node.js, and profile its time on the CPU to get an in-depth view of the execution of the program. Tools also exist that use DTrace’s ability to profile Node.js to create visual representations of the execution stack, such as Brendan Gregg’s Flame Graph Tool.

However, when tracking latency from an application-aware point-of-view, it is necessary to inject probes into the code itself, setting checkpoints, or entry and return points. Using the DTrace Provider Node.js Module written by Chris Andrews, I’ve created a repository of examples on Github showing the DTrace provider in action.

Check out the repository here https://github.com/bahamas10/node-dtrace-examples/.

Read More...