Tricks for Distributed System Debugging and Diagnosis

Distributed systems make for tricky diagnosis problems. Which component is at fault? Is it the network, the machine, the process, or, even worse, some emergent complex behavior? We’ll discuss in depth the tools most commonly used to figure out these thorny issues:

Logs. The perennial go to. We’ll point out unexpected ways to increase logging verbosity at runtime and how to use time-based search across many machines to look at inconsistencies.

Web-based consoles. Must-haves for the modern distributed system, debug servlets can provide surprising insights. We’ll point out what information Hadoop exposes and present easily embeddable open source consoles.

Linux has many goodies in /proc, as well as several tools to look at performance (perf, top). We’ll cover old standbys like lsof and tcpdump, too.

Tracing, when it’s available, can be an incredible tool. HBase recently introduced some, and several other libraries (e.g., Zipkin) have emerged.

Since Hadoop runs on the JVM, we’ll discuss unusual things you can do there, like enabling verbose GC after the fact, poor man’s profiling, and extracting JMX metrics from even the most recalcitrant of processes.

Finally, we’ll bring it all together by doing some simple stats to find and visualize outliers.

The talk will be illustrated by examples from open source systems (especially Hadoop).