For Developers‎ > ‎

Decoding Crash Dumps

This document describes how to process Breakpad minidumps on Linux.

Breakpad tools needed

The tools from Breakpad needed to process crash dumps manually are minidump_stackwalk and dump_syms. It is possible to build these tools from source from within a Chromium checkout on Mac and Linux by running, for example, ninja -C out/Release minidump_stackwalk dump_syms. To build these tools from source in a Breakpad checkout, check out the source from http://code.google.com/p/google-breakpad/ and follow the included instructions. Googlers can use the prebuilt binaries in ~thestig/bin/chrome.

Get the crash dump

Crash dumps (.dmp files) usually come from a crash server i.e. http://crash/ or from the crash reports directory: /path/to/profile/Crash Reports. I.e. ~/.config/google-chrome/Crash Reports/ on Linux.

For Linux crash dumps that are in the crash reports directory, one must strip off the headers before processing it with minidump_stackwalk. Just open the file in a text editor and delete all the lines up until the line that starts with MDMP followed by binary data.

Getting the stacktrace (without symbols)

Run minidump_stackwalk foo.dmp. For 32-bit, minidump_stackwalk will display the stacktrace without symbols. For 64-bit, it will only display the top frame.

Get the debugging symbols

To get symbols or more frames, one needs to have the symbols for the libraries and executables that are part of the stacktrace. Start by running:

minidump_stackwalk foo.dmp /tmp/my_symbols 2>&1 | grep my_symbols

This will print out lines like:
[time stamp] simple_symbol_supplier.cc:150: INFO: No symbol file at /tmp/my_symbols/libfoo/hash/libfoo.sym.

In order to get the symbol file for libfoo, one needs to have a copy of the exact libfoo binary from the system that generated the crash and its corresponding debugging symbols. Oftentimes, Linux distros provide libfoo and its debugging symbols as two separate packages. After obtaining and extracting the packages, use dump_syms to extract the symbols. Assuming the library in question is /lib/libfoo.so and its debugging symbol is /usr/debug/lib/libfoo.so, run:

dump_syms /lib/libfoo.so /usr/debug/lib > /tmp/libfoo.sym

To verify it's the correct version of libfoo, look at the hash from the minidump_stackwalk output and compare it to the hash on the first line. If they match, move /tmp/libfoo.sym to /tmp/my_symbols/libfoo/hash/libfoo.sym and minidump_stackwalk will load it on future runs to give better stacktraces.

Repeat this process for other libraries until minidump_stackwalk outputs the required information.

Decoding Windows crash dumps on Linux

Windows crash dumps can be decoded the same way as Linux crash dumps. The issue is mainly getting the debugging symbols as a .sym file instead of a .pdb file.

To convert a .pdb file to a .sym file:
  1. Obtain the .pdb file and put it on a Windows machine. (It may be possible to do this with Wine, YMMV.)
  2. Download dump_syms.exe.
  3. Run: dump_syms foo.pdb > foo.sym
    • If no error messages, then you are done.
    • If you get: CoCreateInstance CLSID_DiaSource failed (msdia80.dll unregistered?), go to step 4.
  4. Get a copy of msdia80.dll and put it in c:\Program Files\Common Files\Microsoft Shared\VC\.
  5. As Administrator, run: regsrv32 c:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll.
    • On success, retry step 3.
    • If you get error 0x80004005, you did not run as Administrator.

Decoding Mac crash dumps

Comments