The crash reporting system for Google Chrome is called Breakpad, and it integrates into the Chromium source in //components/crash. Crash reporting is disabled by default in Chromium, and crash reports are only available to Google employees, though they are frequently copied into the bug tracker for tracking and fixing.
Breakpad is disabled by default. The cross-platform way to turn on crash reporting is to perform an official, branded build of Chromium by setting GYP_DEFINES="buildtype=Official branding=Chrome", which will require a src-internal checkout.
To enable Breakpad without doing a branded build, each platform has a unique way of enabling breakpad:
Breakpad is not a compile-time option on Windows. Instead, just build the crash_service executable target, which is responsible for catching crashes. Run this before starting chrome.exe with the --noerrdialogs flag.
Breakpad is compiled in, but is not enabled by default. See IsCrashReportingEnabled() in chrome_browser_main_linux.cc for details on when it is enabled.
Set the gyp variable (GYP_DEFINES) linux_dump_symbols=1 to dump the debugging symbols for the chrome binary in the Breakpad symbol format as part of a build.
Set the gyp variable (GYP_DEFINES) mac_breakpad=1 to enable Breakpad support.
After Breakpad is enabled, you can have crash report minidumps generated locally.
To get crash reports for content_shell and layout tests, see this page.
Build and run the crash_service executable target to enable crash catching. Run chrome.exe --noerrdialogs to disable the OS-level error reporter, otherwise it will block the crash_service. Any crashes will then cause a minidump file (.dmp) and metadata sidecar (.txt) to be generated in %HOME%\AppData\Local\Chromium\User Data\Crash Reports.
To make chromium write a .dmp file, run it like
To make chromium write a .dmp file, run it like
Then go to "about:inducebrowsercrashforrealz", and the file is created in ~/Library/Application\ Support/Chromium/Crash\ Reports:
$ ls -l ~/Library/Application\ Support/Chromium/Crash\ Reports/
-rw------- 1 thakis 5000 322016 Oct 11 14:53 ERR.dmp
When a Chrome process crashes, Breakpad springs into action by gathering information about the exception state, callstacks, stack memory, and loaded modules. It takes all of this information and puts it into a minidump file. This minidump is then HTTP POST uploaded to Google, along with some metadata. The metadata contains information like Chrome version, OS name and version, and crash keys.
Once the crash report and metadata are received, a server begins processing it. The first step is to symbolize the instruction pointers to get function names associated with the callstack. The processing servers have platform-neutral Breakpad symbol files for Chrome executables and operating system shared libraries, which allow symbolication of both Chrome functions and OS functions. After symbolizing the crash, the processor generates signatures to help group/bucket crash reports by their cause, implicated by shared stack frames.
The tools used by the processor to symbolize the stack are effectively the same as the minidump_stackwalk and minidump_dump tools available in Breakpad.
Once a crash report is processed, it is made available for view in the crash dashboard. The dashboard allows developers to drill down into crash reports, grouped by metadata. More documentation on this is available internally, linked to from the dashboard. The original minidump file can be downloaded from this frontend, as well.
There are three types of signatures generated by the crash processor:
The stack signature is the topmost symbol name in the crashed thread, suffixed with a small hash of the callstack. This signature provides a high-level grouping and easy identification (as it is human-readable), but it can collide across versions for different complete stack traces. Variance in the crashing thread's callstack can also produce several signatures for the same root cause.
This signature is merely a hash of the crashed stack. This makes it accurate for grouping crashes precisely, but it is not human readable and slight variance in a crashed stack defeat this signature's bucketing properties.
The magic signature is heuristic-based, using application-specific knowledge of the Chromium codebase to find the most germane symbol to produce a signature. The magic signature creator has knowledge of CHECK/DCHECK, hangs caught by the ThreadWatcher watchdog, out-of-memory conditions, memory corruption signs, and other heuristics. This signature is highly simplified, which gives it good bucketing properties and protection against callstack variance, but the signature can often be generated for disjoint crashes from different versions. More information can be found in the processor's design document and this explanation.
The minidump format originated on Windows, and can be natively loaded into a debugger on that platform. Doing this gives you access to local variables, callstacks, and integrates with the source viewer in Visual Studio.
On POSIX systems, however, a minidump is an opaque binary blob. It is possible to examine local variable state, but it requires manual work to do so. To work with minidumps, Breakpad provides two tools (more information here):
minidump_stackwalk prints the stack trace for all of the threads in the minidump. Note that without Breakpad symbol files, placed in a special directory structure, this will not symbolize the stack. It will merely print the %EIP, %EBP, and %ESP (or the x64 equivalent) for each frame and the code module in which the frame resides.
minidump_dump outputs the stack memory for each thread as a hexadecimal string.
Unfortunately, this process is neither straightforward nor quick. But there are alternatives for saving local variables to be visible in a crash report. See below for more information.
However, it may be possible to get most of the same information using WinDBG on Windows. As long as the CPU architecture is supported by Windows, the debugger will be able to uderstand the exception record and disassemble the code around the exception, show call stacks (without symbols though), and let you inspect the contents of the stack and registers. That may be enough to get you going on a good percentage of crashes, especially since the symbolized stack may be available on the crash server.
See also http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/crash-reporting/debugging-a-minidump
To symbolize a minidump that was not uploaded, you can use the crsym tool (internal instance of https://github.com/chromium/crsym/):
To symbolize a minidump for a local build, follow the steps in https://code.google.com/p/chromium/issues/detail?id=304846#c14 for Mac, and see http://www.chromium.org/developers/decoding-crash-dumps for linux.
If a crash is not reproducible, the only tools at one's disposal for investigation are what can be gleaned from a crash report. These are some tips for debugging builds running in production (i.e. on users' machines).
Production builds are highly optimized to make the binary executable fast and small. As part of this, when the compiler is optimizing, it will reuse space on the stack for different variables. To defeat this optimization, you can use base::debug::Alias() on the variable. This will attempt to force the compiler to keep dedicated space around for the variable, so that its value is inspectable in a minidump stackdump.
Another way to make variable values accessible in a crash report is to report it as metadata, rather than keeping it in the minidump. The metadata is viewable in the crash dashboard and is not contained in the minidump file. Learn more about debugging with crash keys here.
Production builds are generally not cut from the tip of tree/trunk, but are branched and stabilized via the dev, beta, and stable release channels. Follow these instructions for how to check out a release branch, which will give you the code for the version of the crash you're investigating.
For Developers >