For Developers‎ > ‎

Crash Reports

The crash reporting system for Google Chrome is called Breakpad, and it integrates into the Chromium source in //components/crash. Crash reporting is disabled by default in Chromium, and crash reports are only available to Google employees, though they are frequently copied into the bug tracker for tracking and fixing.

Enabling Breakpad

Breakpad is disabled by default. The cross-platform way to turn on crash reporting is to perform an official, branded build of Chromium by setting GYP_DEFINES="buildtype=Official branding=Chrome", which will require a src-internal checkout.

To enable Breakpad without doing a branded build, each platform has a unique way of enabling breakpad:

In Chromium

On Windows

Breakpad is not a compile-time option on Windows. Instead, just build the crash_service executable target, which is responsible for catching crashes. Run this before starting chrome.exe with the --noerrdialogs flag.

On Linux

Breakpad is compiled in, but is not enabled by default. See IsCrashReportingEnabled() in chrome_browser_main_linux.cc for details on when it is enabled.

Set the gyp variable (GYP_DEFINES) linux_dump_symbols=1 to dump the debugging symbols for the chrome binary in the Breakpad symbol format as part of a build.

On Mac

Set the gyp variable (GYP_DEFINES) mac_breakpad=1 to enable Breakpad support.

Generating Crash Dumps Locally

After Breakpad is enabled, you can have crash report minidumps generated locally.

To get crash reports for content_shell and layout tests, see this page.

On Windows

Build and run the crash_service executable target to enable crash catching. Run chrome.exe --noerrdialogs to disable the OS-level error reporter, otherwise it will block the crash_service. Any crashes will then cause a minidump file (.dmp) and metadata sidecar (.txt) to be generated in %HOME%\AppData\Local\Chromium\User Data\Crash Reports.

On Linux

TODO: write

On Mac

To make chromium write a .dmp file, run it like

  CHROME_HEADLESS=1 out/Release/Chromium.app/Contents/MacOS/Chromium

Then go to "about:inducebrowsercrashforrealz", and the file is created in ~/Library/Application\ Support/Chromium/Crash\ Reports:

$ ls -l ~/Library/Application\ Support/Chromium/Crash\ Reports/
total 632
-rw-------  1 thakis  5000  322016 Oct 11 14:53 ERR.dmp

Life of a Crash Report

When a Chrome process crashes, Breakpad springs into action by gathering information about the exception state, callstacks, stack memory, and loaded modules. It takes all of this information and puts it into a minidump file. This minidump is then HTTP POST uploaded to Google, along with some metadata. The metadata contains information like Chrome version, OS name and version, and crash keys.

Once the crash report and metadata are received, a server begins processing it. The first step is to symbolize the instruction pointers to get function names associated with the callstack. The processing servers have platform-neutral Breakpad symbol files for Chrome executables and operating system shared libraries, which allow symbolication of both Chrome functions and OS functions. After symbolizing the crash, the processor generates signatures to help group/bucket crash reports by their cause, implicated by shared stack frames.

The tools used by the processor to symbolize the stack are effectively the same as the minidump_stackwalk and minidump_dump tools available in Breakpad.

Once a crash report is processed, it is made available for view in the crash dashboard. The dashboard allows developers to drill down into crash reports, grouped by metadata. More documentation on this is available internally, linked to from the dashboard. The original minidump file can be downloaded from this frontend, as well.

Crash Signatures

There are three types of signatures generated by the crash processor:

Stack Signature

The stack signature is the topmost symbol name in the crashed thread, suffixed with a small hash of the callstack. This signature provides a high-level grouping and easy identification (as it is human-readable), but it can collide across versions for different complete stack traces. Variance in the crashing thread's callstack can also produce several signatures for the same root cause.

Stack Signature 2

This signature is merely a hash of the crashed stack. This makes it accurate for grouping crashes precisely, but it is not human readable and slight variance in a crashed stack defeat this signature's bucketing properties.

Magic Signature

The magic signature is heuristic-based, using application-specific knowledge of the Chromium codebase to find the most germane symbol to produce a signature. The magic signature creator has knowledge of CHECK/DCHECK, hangs caught by the ThreadWatcher watchdog, out-of-memory conditions, memory corruption signs, and other heuristics. This signature is highly simplified, which gives it good bucketing properties and protection against callstack variance, but the signature can often be generated for disjoint crashes from different versions. More information can be found in the processor's design document and this explanation.

Working with Minidumps

The minidump format originated on Windows, and can be natively loaded into a debugger on that platform. Doing this gives you access to local variables, callstacks, and integrates with the source viewer in Visual Studio.

On POSIX systems, however, a minidump is an opaque binary blob. It is possible to examine local variable state, but it requires manual work to do so. To work with minidumps, Breakpad provides two tools (more information here):

minidump_stackwalk prints the stack trace for all of the threads in the minidump. Note that without Breakpad symbol files, placed in a special directory structure, this will not symbolize the stack. It will merely print the %EIP, %EBP, and %ESP (or the x64 equivalent) for each frame and the code module in which the frame resides.

minidump_dump outputs the stack memory for each thread as a hexadecimal string.

Extracting Local Variables on POSIX

This is one workflow for manually analyzing a minidump:
  1. Run minidump_dump and minidump_stackwalk on the target minidump file, piping output to text files.
  2. Find the target thread in the stackwalk (via the threadid).
  3. Find the corresponding target thread in the stackdump.
  4. Take the raw stack data for that thread, and remove the 0x prefix.
  5. Split the stack data on word-size boundaries (8 hexadecimal characters on a 32-bit platform).
  6. Using the information from the stackwalk, annotate EIP, EBP, and ESP in the stackdump to delineate stack frames. Note that the byte order/endianness in the stackdump may not match host order, i.e. the pointer badbeef may be represented in the minidump as efbedbba.
  7. Download the build products (executable and symbol file) for the version of Chrome you're working with. Disassemble the function you're interested in.
  8. Step through the assembly and see how data is being pushed and popped off the stack, and try to correlate it to the data in the stackdump.
Unfortunately, this process is neither straightforward nor quick. But there are alternatives for saving local variables to be visible in a crash report. See below for more information.

However, it may be possible to get most of the same information using WinDBG on Windows. As long as the CPU architecture is supported by Windows, the debugger will be able to uderstand the exception record and disassemble the code around the exception, show call stacks (without symbols though), and let you inspect the contents of the stack and registers. That may be enough to get you going on a good percentage of crashes, especially since the symbolized stack may be available on the crash server.

Symbolizing Minidumps

To symbolize a minidump that was not uploaded, you can use the crsym tool (internal instance of https://github.com/chromium/crsym/):
  1. Build minidump_stackwalk from inside the Chromium or Breakpad trees.
  2. Run mindiump_stackwalk -m /path/to/minidump.dmp > /tmp/stack.txt
  3. Go to crsym and set the input type to be a Minidump Stackwalk.
  4. Paste the contents of /tmp/stack.txt into the input field.
  5. Press the Symbolize button and wait for the output.
To symbolize a minidump for a local build, follow the steps in https://code.google.com/p/chromium/issues/detail?id=304846#c14 for Mac, and see http://www.chromium.org/developers/decoding-crash-dumps for linux.

Production Debugging

If a crash is not reproducible, the only tools at one's disposal for investigation are what can be gleaned from a crash report. These are some tips for debugging builds running in production (i.e. on users' machines).

Aliasing Against Optimizations

Production builds are highly optimized to make the binary executable fast and small. As part of this, when the compiler is optimizing, it will reuse space on the stack for different variables. To defeat this optimization, you can use base::debug::Alias() on the variable. This will attempt to force the compiler to keep dedicated space around for the variable, so that its value is inspectable in a minidump stackdump.

Crash Keys

Another way to make variable values accessible in a crash report is to report it as metadata, rather than keeping it in the minidump. The metadata is viewable in the crash dashboard and is not contained in the minidump file. Learn more about debugging with crash keys here.

Checking Out a Release Branch

Production builds are generally not cut from the tip of tree/trunk, but are branched and stabilized via the dev, beta, and stable release channels. Follow these instructions for how to check out a release branch, which will give you the code for the version of the crash you're investigating.
Comments