Chrome is client-side software, which means that sometimes there are bugs that can occur only on users' machines ("in production") that cannot be reproduced by QA or engineering. When this happens, it's often helpful to gather bug-specific data from production to help pinpoint the cause of the crash. The crash key logging system is a generified method to help do that.
The core of the crash key logging system is in base/debug/crash_logging.h, which allows crash keys to be added in virtually any module. At the most basic level, it is a system that provides a map structure to record key-value pairs. These pairs are uploaded as POST form-multipart data when Breakpad uploads a minidump to the Google crash reporting system. (The data therefore is only accessible to those with access to crash reports internally at Google). Before a key can be used, it must be registered in the list of allowed crash keys, specified at chrome/common/crash_keys.h. Pre-registration is required so that space can be allocated for it prior to any crash happening, since in a crashed process context, dynamic memory allocation routines are not necessarily safe to use. Finally, the platform-specific Breakpad implementations each have a way to record, without allocating, the key and value pairs when they are set using crash_logging.h.
The crash key system is used to report some common pieces of data, as well as things that happen in exceptional cases: the URL of the webpage, command line switches, active extension IDs, GPU vendor information, experiment/variations information, etc. All this metadata is set, stored, and reported using the crash logging system described here.
Imagine you are investigating a crash, and you want to know the value of some variable when the crash occurs; the crash key logging system enables you to do just that.
In chrome/common/crash_keys.h, define a new
There are three size classes for registering keys:
Now that the key has been registered, it can be used in code to gather data from production. The routines are defined in chrome/common/crash_keys.h and the most basic usage is
In addition, there is
Using http://go/crash (internal only), find the crash report signature related to your bug, and click on the "N of M" reports link to drill down to report-specific information. From there, select a report and go to the "Fields" tab to view all the crash key-value pairs.
Now imagine a scenario where you have a use-after-free. The crash reports coming in do not indicate where the object being used was initially freed, however, just where it is later being dereferenced. To make debugging easier, it would be nice to have the stack trace of the destructor, and the crash key system works for that, too.
As above, define and register a crash key with the system. If you expect a deep stack trace, you may want to use
To set a stack trace to a crash key, use one of the specialized routines from crash_logging.h. This will typically be done like so:
Unlike with the previous example, a stack trace will just be a string of hexadecimal addresses. To turn the addresses back into symbols use, http://go/crsym (internal instance of https://github.com/chromium/crsym/). Using the Crash Key input type, give it a crash report ID and the name of your crash key. Crsym will then fetch the symbol data from the internal crash processing backends and return a formatted, symbolized stack trace.