The primary purpose of ClusterFuzz is to find bugs by analyzing crashes. The test case list page provides information on recent crashes and allows users to search based on certain criteria.
By default, when typing into the search box ClusterFuzz will filter based on all displayed information about the test case as well as some additional metadata, such as the name of the fuzzer that found the crash. The general formats for searches by a field name are “field:value” or “field=value”. If you wish to search for a string that may contain spaces, enclose the value you want to search for in quotes. A full list of search fields can be found in the advanced options section.
ClusterFuzz will filter the list to only show fields that match all of your criteria. If you wish to list terms that match any of a number of criteria, separate the terms with “or” (e.g. “impacts:stable or impacts:beta”).
If you have taken the time to write a fuzzer for ClusterFuzz, you probably want to know when it finds crashes. Luckily, this is very easy. Simply type the name of your fuzzer into the search box, and browse the listed test cases.
While triaging incoming bugs from ClusterFuzz, it is often useful to search the list for all confirmed (non-flaky) security bugs. To do this, simply enter “confirmed:yes security:yes” into the search box. This works by filtering the list to only display test cases that match both of those criteria. If you wanted to search for all confirmed security bugs found by a fuzzer named “test_fuzzer”, you could search for “confirmed:yes security:yes test_fuzzer”.
It can also be useful to display only crashes that do not yet have bugs filed for them. ClusterFuzz supports the keyword “null” for value searches as an alias for the empty string. You can search for test cases that do not have any bugs filed for them by enter “bug_information:null” into the search box (this is equivalent to a search for bug_information:””).
If you wish to see additional information about all test cases listed, you can click the “Switch to detailed view” link above the search box in the top-right corner of the test case list page. Additional information includes the name of the fuzzer that found the crash, job type, original file name of the test case, and whether or not it is a stable regression. It also provides access to a few options that make it easy to perform bulk operations on test cases, such as redoing jobs and deletion.
The previous examples do not cover all cases. A full list of fields that you can search by are included below.
Once you have identified a test case that you wish to investigate further, ClusterFuzz will provide you with detailed information on the crash. It also simplifies some common tasks that you may want to perform, such as filing a bug.
To open a new bug from the test case details page, simply click on the “file bug” button at the top of the page, fill out any information you can, and submit the form. For security bugs, you will also be prompted to select the severity. For guidance on how to determine the severity of a bug, see the chromium severity guidelines.
Similarly, you can use the “update bug” button to associate a test case with an existing bug. To do this, you only need to provide the bug number that it should be associated with.
Sometimes, ClusterFuzz messes up. This is most common with flaky test cases and test cases that take a long time to process. If ClusterFuzz reports information that you suspect is wrong for a particular test case, such as a poorly minimized test case or incorrect regression range, you can redo certain jobs by clicking the “redo” button. Check the boxes for the jobs you wish to redo, and submit the form.
If for any reason it makes sense to delete a test case, this can be done from the test case details page. Use the “delete” button and follow the prompts to delete the test case if you are sure that it should be deleted.
For job types that do not use custom builds, frames in the stack trace will link directly to the relevant chromium code in Code Search. This can be useful when analyzing the crash to try to triage it, or for initial investigation if you are assigned a ClusterFuzz bug.
Whenever possible, ClusterFuzz will attempt to calculate a range of Chromium revisions where a crash was introduced. This is referred to as the regression range. This is not attempted when a test case is flaky, or when a custom binary is being used by the job type for the test case.
If you click on the link for a range, you will be taken to a page that lists not just the revision range for chrome, but also for dependencies such as Blink, V8, NaCl, and more. For the chromium revision and certain commonly updated dependencies, such as Blink, there will be links directly to the changelist for the revision range in question. For less commonly updated dependences, you may need to manually find the changelist.
Occasionally, ClusterFuzz test cases will have special requirements that must be taken into consideration while trying to reproduce a test case.
Test cases that require HTTP must be hosted on an HTTP server to be reproduced.
Tests that require interaction gestures will be accompanied by an interaction gesture string. This string was automatically generated by ClusterFuzz and may include various keyboard inputs, mouse movements and clicks, or other gestures. Sometimes, the gesture causing the crash may be obvious, though other times this can be difficult to reproduce these crashes outside of ClusterFuzz.
ClusterFuzz does much more than just fuzzing. When crashes are detected, it performs additional tasks such as test case minimization, finding regression ranges, determining whether a bug affects the current stable and beta branches, checking to see if the bug has been fixed, and more. Obviously, all of these things can be useful even for bugs that were not found by fuzzers running in ClusterFuzz. The test case uploads page allows you to upload test cases to ClusterFuzz for analysis.
To begin uploading a test case, navigate to the test uploads tab in ClusterFuzz and select a file to upload. If your test case requires multiple files, you can upload an archive. If you do this, make sure that the file that should be passed to chrome on the command line contains the string “run” or “index” somewhere in its filename. Single file uploads will be passed to chrome regardless of file name.
Sometimes, it makes sense to upload multiple test cases to ClusterFuzz at once for simple verification and analysis. To do this, bundle the test cases into a zip, tar.gz, or tar.bz2 archive and select the archive as the file to upload. Check the “Multiple test cases” checkbox and continue the upload using the same process used for a single test case. Any other settings, such as job type, will be applied when testing all files in the archive.
The “raw” option allows test cases to be uploaded by providing the contents of a file directly instead of uploading a file. This is useful when dealing with reports that have very short reproduction cases, as reporters will usually just paste the contents directly into the report body in these cases.
Occasionally, it makes sense to have ClusterFuzz create a test case from a URL rather than a file. This can be useful if crash reports are coming in for a particular URL and you want to test it quickly. Simply enter the URL into the text box, and the bots will pull down a copy of the current page contents. This does not always work consistently, so if it fails you may need to take some time to prepare a test case to upload rather than using this option.
If you enter a bug number, ClusterFuzz will automatically update that bug once it has finished processing the test case. When ClusterFuzz detects that the issue has been fixed, it will update the bug again with the fixed range.
Time in seconds to run this test case before giving up and terminating the binary. This defaults to whatever the specified timeout is for your selected job type. Only change it to a higher value for a flaky test where you think a larger timeout will help. ClusterFuzz tries each test many times before determining that it is unreproducible, so selecting an excessively high value here will usually only result in a long wait before receiving a report.
ClusterFuzz generates random user gestures while fuzzing test cases. To be able to reproduce these simulated gestures, it associates a list of them with each test case. If you want to try to reproduce a test case with a ClusterFuzz gesture string, you can provide one when uploading the test case.
This option should be used if the bot should host the test case on a local HTTP server rather than passing it to chrome directly on the command line. This is not usually necessary, but may be required for certain test cases.
A few of the bots that make up ClusterFuzz are more powerful than the others. They are used to perform some specific tasks such as symbolization. Some test cases may not be able to run properly on a lower-end bot. In these cases, you can choose to analyze the test case on a high-end bot. Please make sure that you only do this when it is absolutely necessary. Try running on a lower-end bot first, and reupload with the high-end bot option only if it did not reproduce and you believe that this will make a difference.
As the name suggests, fuzzing is the primary purpose of ClusterFuzz. For more information on writing a fuzzer and preparing it for use with ClusterFuzz, see the ClusterFuzz fuzzer guide.
If your fuzzer has been prepared according to the fuzzer guide, the actual upload process is fairly straightforward. From the fuzzers page, you can fill out the fuzzer upload form. Only a name and the archive that makes up your fuzzer are required.
The fuzzer name must be unique. If you give your fuzzer the same name as another fuzzer, it will update the existing fuzzer to a new version. This is handled this way because, occasionally, advanced options must be changed from fuzzer version to version. Many advanced options are not included in the simple “update fuzzer” form in the fuzzer list. In general, it is a good idea to give your fuzzers a simple name prefixed with something that identifies you.
Next, you should select the archive to upload for your fuzzer. ClusterFuzz supports zip, tar.gz, and tar.bz2 uploads. As explained in the fuzzer guide, it is best to include the string “run” somewhere in the name of the file that should actually be executed to run your fuzzer. Doing this will allow ClusterFuzz to detect it automatically. If you have more than one file with “run” in its name, or if for some reason you are unable to name the file to execute your fuzzer in this way, you can specify the name by explicitly entering an “executable path” in the fuzzer upload form.
You will also have to provide job types that should be used for your fuzzer. Job types are defined on the jobs page and control factors such as which platform the test cases should be processed on, which binary should be used for fuzzing, and much more. The most commonly used job type is “linux_asan_chrome_mp”, which simply passes the test cases to an asanified chromium build on a Linux bot.
Usually, to update a fuzzer you simply need to select it in the fuzzer list, select “update fuzzer”, and upload the new file. If the update requires a change to one of the advanced options for the fuzzer, upload it as though it were a new fuzzer and use the same name. After you update the fuzzer, you should check that it is running properly by making sure it is still generating the correct number of test cases by viewing test run output in the test case list. Unless you have a good reason to do so, you should not delete your fuzzer. This can cause some issues for existing test cases associated with your fuzzer.
You can upload a data bundle for a fuzzer from the fuzzer list page. To do this, find the fuzzer in the fuzzer list either by paging through it or searching for the fuzzer name. When you have identified the fuzzer that you want to add a data bundle for, simply click on the update data bundle button, select the archive for the data bundle, and upload it. This does not support any advanced data bundle options. To use these options, the data bundle will have to be uploaded from the data bundles page. See the uploading data bundles section for more information.
Occasionally fuzzers require special configuration. This is most common for fuzzers that test network protocols and require scripts to run not just when fuzzing, but also when Chrome is launched.
Time, in seconds, to wait before terminating the binary and assuming that there was no crash. This should only be modified if you have a very good reason to do so. Inflating the timeout unnecessarily wastes cycles on the bots.
Number of test cases to generate each time the fuzzer runs. In most cases the default is 1000, though this varies by job type.
Path within the archive of the binary to run for fuzzing. It must accept the standard command line arguments for ClusterFuzz, as mentioned in the fuzzer writing guide.
New fuzzers should not use background scripts. Use launcher scripts instead. Background scripts are arbitrary scripts that are launched before any test case associated with a particular fuzzer runs.
The script must have some level of awareness as to how to find test cases generated by the fuzzer, and test cases generated by the fuzzer must have a way to redirect Chrome to a resource managed by the background script. For example, a fuzzer may generate a file named fuzz-deprecated-old-fuzzer-1.html that redirects to localhost:9999/resource.blah, and a file named resource.blah. The background script could listen on port 9999 and do something with resource.blah to generate a response.
Launcher scripts are scripts that are executed instead of directly launching Chrome when processing fuzzed test cases. Arguments passed to the launcher script will make up the command that should be executed for the job type, followed by the resource generated by the fuzzer. See the launcher scripts section of the fuzzer guide for more information.
The external contribution checkbox must be checked if you are uploading a fuzzer from an external contributor. This makes it easy to keep track of reports from the fuzzer that may be eligible for rewards under the Chromium Vulnerability Rewards Program.
Many fuzzers rely on some kind of input data to generate test cases. The data associated with a fuzzer is separated into a data bundle so that either one can be updated easily without interfering with the other. It also allows certain data bundles to be shared between multiple fuzzers. For small data sets (less than 100MB), no special configuration is required. The “Cloud Storage” option should be used for larger data bundles.
Basic data bundles can be uploaded directly from the fuzzer list page as described in the uploading fuzzers section. They can also be uploaded by filling out the form on the data bundle list page. When using the data bundle upload form, the fuzzer name must be specified. If your data bundle is small, do not check the “use Google Cloud Storage” checkbox. For more information, or if you are uploading a large data bundle, see uploading a Cloud Storage data bundle.
The process for uploading a Cloud Storage data bundle is essentially the same as uploading a basic data bundle, but the underlying behavior is very different. Disk space on the ClusterFuzz bots is limited, so large data bundles are stored on Cloud Storage and replicated on certain bots in each zone that serve these files using glusterfs. Accessing these files is slower, and unlike the directories for usual data bundles, these are read-only.
Crash statistics serve multiple purposes in ClusterFuzz. If you do not have a particular fuzzer in mind while viewing the statistics, you can see the most common crash stacks and frequency with which they are being triggered. If you are developing a new fuzzer, you can confirm that it is working properly.
To view a list of all crashes sorted by frequency, simply visit the crash statistics page. From here, you can filter using the list using the search box and order by any of the columns in the table.
Several options are provided for filtering crash statistics. This can be useful if you only want to see recent crashes, crashes for a particular fuzzer, or have other criteria that you want to filter by.
If you only want to see crashes found while running a particular fuzzer, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found by the selected fuzzer while using the selected job type.
If you only want to see crashes found while running a particular job type, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found for the selected fuzzer while using the selected job type.
You can also choose to limit the list to crashes with security implications. By default, all crashes are shown.
By default, the crash statistics page only shows crashes from the previous day. If you want to search farther back in time, you can change this here.
ClusterFuzz allows job types to be defined easily. A job definition is a collection of environment variables set on the bots running a job. These define which binary to run, how many test cases to generate, and much more.
Sometimes, managing a custom binary is not ideal. This usually happens if you are testing an area of the code that changes frequently enough that you do not want to manage a custom binary, but also cannot pass test cases to chrome directly. This most commonly comes up when working with IPC handlers.
Often, fuzzer test cases cannot be passed to Chrome directly. Whenever possible, a build target should be added so that ClusterFuzz can pick up changes automatically, and so that it can do additional analysis such as regression or fixed testing.
Job types for builds from this location should include the following definitions:
RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-release-([0-9]+).zip
SYM_RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-symbolized-linux-release-([0-9]+).zip
SYM_DEBUG_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-debug-([0-9]+).zip
REVISION_PATTERN = .*-([0-9]+).zip
STABLE_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-stable-([0-9.]+).zip
BETA_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-beta-([0-9.]+).zip
VERSION_PATTERN = .*-([0-9.]+).zip
These tell ClusterFuzz where to pull its builds from. You will then need to define an APP_NAME, which is the name of the binary within the archive that should be executed, and COMMAND_FORMAT, which represents how the command line for running your application should be formed.
APP_NAME = my_target
COMMAND_FORMAT = %TESTCASE%
The previous example is the simplest case possible and would launch a command of the form “/path/to/my_target fuzz-mytestcase-1”. You can add as many command line arguments as you need. If your test case is passed as a variable, COMMAND_FORMAT can be defined as something like the following:
COMMAND_FORMAT = --flag-1 --testcase-file=%TESTCASE%
Using a custom binary is much simpler than working with an asanified chrome target, but has the disadvantage of needing to be manually updated. When defining the job type, simply upload an archive along with the job definition. This will be the custom binary associated with the job. The name of the binary itself should be specified by the APP_NAME variable, and the format for the command should be specified by the COMMAND_FORMAT variable. “CUSTOM_BINARY = True” must also be defined for job types that use custom binaries, but this will be added automatically.
A sample job definition that uses a custom binary follows:
APP_NAME = chrome
COMMAND_FORMAT = --snipped-some-arguments --ppapi-flash-path=%APP_DIR%/libpepflashplayer.so --user-data-dir=%USER_PROFILE_DIR% %TESTCASE%
USER_PROFILE_ARG = --user-data-dir
NET_LOG_ARG = --log-net-log
WINDOW_ARG = --window-size=$WIDTH,$HEIGHT --window-position=$LEFT,$TOP
MAX_FUZZ_THREADS = 4
TEST_TIMEOUT = 18
MULTI_PROCESS = True
CUSTOM_BINARY = True
A job definition is made up of a set of variables that will be defined as environment variables when ClusterFuzz runs. Some commonly used ones are included below, but this is not intended to be a complete list.
You may have noticed that COMMAND_FORMAT supports some strings with special meanings, such as %TESTCASE%. A full list of these is included in the following table.