This document describes how to process Breakpad minidumps on Linux.
The tools from Breakpad needed to process crash dumps manually are minidump_stackwalk and dump_syms. It is possible to build these tools from source from within a Chromium checkout on Mac and Linux by running, for example, ninja -C out/Release minidump_stackwalk dump_syms. To build these tools from source in a Breakpad checkout, check out the source from http://code.google.com/p/google-breakpad/ and follow the included instructions. Prebuilt binaries from thestig@ are available for download.
Crash dumps (.dmp files) usually come from a crash server i.e. http://crash/ or from the crash reports directory: /path/to/profile/Crash Reports. I.e. ~/.config/google-chrome/Crash Reports/ on Linux.
For Linux crash dumps that are in the crash reports directory, one must strip off the headers before processing it with minidump_stackwalk. Just open the file in a text editor and delete all the lines up until the line that starts with MDMP followed by binary data.
Run minidump_stackwalk foo.dmp. For 32-bit, minidump_stackwalk will display the stacktrace without symbols. For 64-bit, it will only display the top frame.
To get symbols or more frames, one needs to have the symbols for the libraries and executables that are part of the stacktrace. Start by running:
minidump_stackwalk foo.dmp /tmp/my_symbols 2>&1 | grep my_symbols
This will print out lines like:
[time stamp] simple_symbol_supplier.cc:150: INFO: No symbol file at /tmp/my_symbols/libfoo/hash/libfoo.sym.
In order to get the symbol file for libfoo, one needs to have a copy of the exact libfoo binary from the system that generated the crash and its corresponding debugging symbols. Oftentimes, Linux distros provide libfoo and its debugging symbols as two separate packages. After obtaining and extracting the packages, use dump_syms to extract the symbols. Assuming the library in question is /lib/libfoo.so and its debugging symbol is /usr/debug/lib/libfoo.so, run:
dump_syms /lib/libfoo.so /usr/debug/lib > /tmp/libfoo.sym
To verify it's the correct version of libfoo, look at the hash from the minidump_stackwalk output and compare it to the hash on the first line. If they match, move /tmp/libfoo.sym to /tmp/my_symbols/libfoo/hash/libfoo.sym and minidump_stackwalk will load it on future runs to give better stacktraces.
Repeat this process for other libraries until minidump_stackwalk outputs the required information.
Windows crash dumps can be decoded the same way as Linux crash dumps. The issue is mainly getting the debugging symbols as a .sym file instead of a .pdb file.
To convert a .pdb file to a .sym file:
If you've built Chromium.app with symbols, the easiest way to symbolize a crash is to let Crashpad forward the crash to the system crash reporter. See set_system_crash_reporter_forwarding.
For Developers >