Searching the test case listThe primary purpose of ClusterFuzz is to find bugs by analyzing crashes. The test case list page provides information on recent crashes and allows users to search based on certain criteria. By default, when typing into the search box ClusterFuzz will filter based on all displayed information about the test case as well as some additional metadata, such as the name of the fuzzer that found the crash. The general formats for searches by a field name are “field:value” or “field=value”. If you wish to search for a string that may contain spaces, enclose the value you want to search for in quotes. A full list of search fields can be found in the advanced options section. ClusterFuzz will filter the list to only show fields that match all of your criteria. If you wish to list terms that match any of a number of criteria, separate the terms with “or” (e.g. “impacts:stable or impacts:beta”). List bugs from a particular fuzzerIf you have taken the time to write a fuzzer for ClusterFuzz, you probably want to know when it finds crashes. Luckily, this is very easy. Simply type the name of your fuzzer into the search box, and browse the listed test cases. List confirmed security bugsWhile triaging incoming bugs from ClusterFuzz, it is often useful to search the list for all confirmed (non-flaky) security bugs. To do this, simply enter “confirmed:yes security:yes” into the search box. This works by filtering the list to only display test cases that match both of those criteria. If you wanted to search for all confirmed security bugs found by a fuzzer named “test_fuzzer”, you could search for “confirmed:yes security:yes test_fuzzer”. List crashes without associated bugsIt can also be useful to display only crashes that do not yet have bugs filed for them. ClusterFuzz supports the keyword “null” for value searches as an alias for the empty string. You can search for test cases that do not have any bugs filed for them by enter “bug_information:null” into the search box (this is equivalent to a search for bug_information:””). Using detailed viewIf you wish to see additional information about all test cases listed, you can click the “Switch to detailed view” link above the search box in the top-right corner of the test case list page. Additional information includes the name of the fuzzer that found the crash, job type, original file name of the test case, and whether or not it is a stable regression. It also provides access to a few options that make it easy to perform bulk operations on test cases, such as redoing jobs and deletion. Advanced optionsThe previous examples do not cover all cases. A full list of fields that you can search by are included below.
Viewing test case detailsOnce you have identified a test case that you wish to investigate further, ClusterFuzz will provide you with detailed information on the crash. It also simplifies some common tasks that you may want to perform, such as filing a bug. Issue tracker optionsTo open a new bug from the test case details page, simply click on the “file bug” button at the top of the page, fill out any information you can, and submit the form. For security bugs, you will also be prompted to select the severity. For guidance on how to determine the severity of a bug, see the chromium severity guidelines. Similarly, you can use the “update bug” button to associate a test case with an existing bug. To do this, you only need to provide the bug number that it should be associated with.
Redo analysis for a bugSometimes, ClusterFuzz messes up. This is most common with flaky test cases and test cases that take a long time to process. If ClusterFuzz reports information that you suspect is wrong for a particular test case, such as a poorly minimized test case or incorrect regression range, you can redo certain jobs by clicking the “redo” button. Check the boxes for the jobs you wish to redo, and submit the form. Delete a test caseIf for any reason it makes sense to delete a test case, this can be done from the test case details page. Use the “delete” button and follow the prompts to delete the test case if you are sure that it should be deleted. Code Search integrationFor job types that do not use custom builds, frames in the stack trace will link directly to the relevant chromium code in Code Search. This can be useful when analyzing the crash to try to triage it, or for initial investigation if you are assigned a ClusterFuzz bug. Regression and fixed analysisWhenever possible, ClusterFuzz will attempt to calculate a range of Chromium revisions where a crash was introduced. This is referred to as the regression range. This is not attempted when a test case is flaky, or when a custom binary is being used by the job type for the test case. If you click on the link for a range, you will be taken to a page that lists not just the revision range for chrome, but also for dependencies such as Blink, V8, NaCl, and more. For the chromium revision and certain commonly updated dependencies, such as Blink, there will be links directly to the changelist for the revision range in question. For less commonly updated dependences, you may need to manually find the changelist.
Additional requirementsOccasionally, ClusterFuzz test cases will have special requirements that must be taken into consideration while trying to reproduce a test case. HTTPTest cases that require HTTP must be hosted on an HTTP server to be reproduced. Interaction gesturesTests that require interaction gestures will be accompanied by an interaction gesture string. This string was automatically generated by ClusterFuzz and may include various keyboard inputs, mouse movements and clicks, or other gestures. Sometimes, the gesture causing the crash may be obvious, though other times this can be difficult to reproduce these crashes outside of ClusterFuzz. Uploading test casesClusterFuzz does much more than just fuzzing. When crashes are detected, it performs additional tasks such as test case minimization, finding regression ranges, determining whether a bug affects the current stable and beta branches, checking to see if the bug has been fixed, and more. Obviously, all of these things can be useful even for bugs that were not found by fuzzers running in ClusterFuzz. The test case uploads page allows you to upload test cases to ClusterFuzz for analysis. Uploading a single test case from a fileTo begin uploading a test case, navigate to the test uploads tab in ClusterFuzz and select a file to upload. If your test case requires multiple files, you can upload an archive. If you do this, make sure that the file that should be passed to chrome on the command line contains the string “run” or “index” somewhere in its filename. Single file uploads will be passed to chrome regardless of file name.
Uploading multiple test cases at onceSometimes, it makes sense to upload multiple test cases to ClusterFuzz at once for simple verification and analysis. To do this, bundle the test cases into a zip, tar.gz, or tar.bz2 archive and select the archive as the file to upload. Check the “Multiple test cases” checkbox and continue the upload using the same process used for a single test case. Any other settings, such as job type, will be applied when testing all files in the archive.
Uploading a “raw” test caseThe “raw” option allows test cases to be uploaded by providing the contents of a file directly instead of uploading a file. This is useful when dealing with reports that have very short reproduction cases, as reporters will usually just paste the contents directly into the report body in these cases.
Uploading a “URL” test caseOccasionally, it makes sense to have ClusterFuzz create a test case from a URL rather than a file. This can be useful if crash reports are coming in for a particular URL and you want to test it quickly. Simply enter the URL into the text box, and the bots will pull down a copy of the current page contents. This does not always work consistently, so if it fails you may need to take some time to prepare a test case to upload rather than using this option. Advanced optionsBug informationIf you enter a bug number, ClusterFuzz will automatically update that bug once it has finished processing the test case. When ClusterFuzz detects that the issue has been fixed, it will update the bug again with the fixed range. TimeoutTime in seconds to run this test case before giving up and terminating the binary. This defaults to whatever the specified timeout is for your selected job type. Only change it to a higher value for a flaky test where you think a larger timeout will help. ClusterFuzz tries each test many times before determining that it is unreproducible, so selecting an excessively high value here will usually only result in a long wait before receiving a report. GesturesClusterFuzz generates random user gestures while fuzzing test cases. To be able to reproduce these simulated gestures, it associates a list of them with each test case. If you want to try to reproduce a test case with a ClusterFuzz gesture string, you can provide one when uploading the test case. HTTP supportThis option should be used if the bot should host the test case on a local HTTP server rather than passing it to chrome directly on the command line. This is not usually necessary, but may be required for certain test cases. High-end botA few of the bots that make up ClusterFuzz are more powerful than the others. They are used to perform some specific tasks such as symbolization. Some test cases may not be able to run properly on a lower-end bot. In these cases, you can choose to analyze the test case on a high-end bot. Please make sure that you only do this when it is absolutely necessary. Try running on a lower-end bot first, and reupload with the high-end bot option only if it did not reproduce and you believe that this will make a difference. Uploading fuzzersAs the name suggests, fuzzing is the primary purpose of ClusterFuzz. For more information on writing a fuzzer and preparing it for use with ClusterFuzz, see the ClusterFuzz fuzzer guide. Uploading a simple fuzzerIf your fuzzer has been prepared according to the fuzzer guide, the actual upload process is fairly straightforward. From the fuzzers page, you can fill out the fuzzer upload form. Only a name and the archive that makes up your fuzzer are required. The fuzzer name must be unique. If you give your fuzzer the same name as another fuzzer, it will update the existing fuzzer to a new version. This is handled this way because, occasionally, advanced options must be changed from fuzzer version to version. Many advanced options are not included in the simple “update fuzzer” form in the fuzzer list. In general, it is a good idea to give your fuzzers a simple name prefixed with something that identifies you. Next, you should select the archive to upload for your fuzzer. ClusterFuzz supports zip, tar.gz, and tar.bz2 uploads. As explained in the fuzzer guide, it is best to include the string “run” somewhere in the name of the file that should actually be executed to run your fuzzer. Doing this will allow ClusterFuzz to detect it automatically. If you have more than one file with “run” in its name, or if for some reason you are unable to name the file to execute your fuzzer in this way, you can specify the name by explicitly entering an “executable path” in the fuzzer upload form. You will also have to provide job types that should be used for your fuzzer. Job types are defined on the jobs page and control factors such as which platform the test cases should be processed on, which binary should be used for fuzzing, and much more. The most commonly used job type is “linux_asan_chrome_mp”, which simply passes the test cases to an asanified chromium build on a Linux bot. Updating a fuzzer to a new versionUsually, to update a fuzzer you simply need to select it in the fuzzer list, select “update fuzzer”, and upload the new file. If the update requires a change to one of the advanced options for the fuzzer, upload it as though it were a new fuzzer and use the same name. After you update the fuzzer, you should check that it is running properly by making sure it is still generating the correct number of test cases by viewing test run output in the test case list. Unless you have a good reason to do so, you should not delete your fuzzer. This can cause some issues for existing test cases associated with your fuzzer. Uploading a data bundleYou can upload a data bundle for a fuzzer from the fuzzer list page. To do this, find the fuzzer in the fuzzer list either by paging through it or searching for the fuzzer name. When you have identified the fuzzer that you want to add a data bundle for, simply click on the update data bundle button, select the archive for the data bundle, and upload it. This does not support any advanced data bundle options. To use these options, the data bundle will have to be uploaded from the data bundles page. See the uploading data bundles section for more information. Advanced optionsOccasionally fuzzers require special configuration. This is most common for fuzzers that test network protocols and require scripts to run not just when fuzzing, but also when Chrome is launched. Test case timeoutTime, in seconds, to wait before terminating the binary and assuming that there was no crash. This should only be modified if you have a very good reason to do so. Inflating the timeout unnecessarily wastes cycles on the bots. Test case countNumber of test cases to generate each time the fuzzer runs. In most cases the default is 1000, though this varies by job type. Executable pathPath within the archive of the binary to run for fuzzing. It must accept the standard command line arguments for ClusterFuzz, as mentioned in the fuzzer writing guide. Launcher scriptLauncher scripts are scripts that are executed instead of directly launching Chrome when processing fuzzed test cases. Arguments passed to the launcher script will make up the command that should be executed for the job type, followed by the resource generated by the fuzzer. See the launcher scripts section of the fuzzer guide for more information. External contributionThe external contribution checkbox must be checked if you are uploading a fuzzer from an external contributor. This makes it easy to keep track of reports from the fuzzer that may be eligible for rewards under the Chromium Vulnerability Rewards Program. Uploading data bundlesMany fuzzers rely on some kind of input data to generate test cases. The data associated with a fuzzer is separated into a data bundle so that either one can be updated easily without interfering with the other. It also allows certain data bundles to be shared between multiple fuzzers. For small data sets (less than 100MB), no special configuration is required. The “Cloud Storage” option should be used for larger data bundles. Uploading a basic data bundle (< 100MB)Basic data bundles can be uploaded directly from the fuzzer list page as described in the uploading fuzzers section. They can also be uploaded by filling out the form on the data bundle list page. When using the data bundle upload form, the fuzzer name must be specified. If your data bundle is small, do not check the “use Google Cloud Storage” checkbox. For more information, or if you are uploading a large data bundle, see uploading a Cloud Storage data bundle.
Uploading a Cloud Storage data bundle (> 100MB)The process for uploading a Cloud Storage data bundle is essentially the same as uploading a basic data bundle, but the underlying behavior is very different. Disk space on the ClusterFuzz bots is limited, so large data bundles are stored on Cloud Storage and replicated on certain bots in each zone that serve these files using glusterfs. Accessing these files is slower, and unlike the directories for usual data bundles, these are read-only. LibFuzzer corporaSee https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/efficient_fuzzer.md#Seed-Corpus for information of how to upload a seed corpus for a LibFuzzer-based fuzzer. There is more automation in place for libFuzzer fuzzers, so you don't need to go through the usual ClusterFuzz data bundle flow. Viewing crash statisticsCrash statistics serve multiple purposes in ClusterFuzz. If you do not have a particular fuzzer in mind while viewing the statistics, you can see the most common crash stacks and frequency with which they are being triggered. If you are developing a new fuzzer, you can confirm that it is working properly. Viewing all crashesTo view a list of all crashes sorted by frequency, simply visit the crash statistics page. From here, you can filter using the list using the search box and order by any of the columns in the table. Filtering crash statisticsSeveral options are provided for filtering crash statistics. This can be useful if you only want to see recent crashes, crashes for a particular fuzzer, or have other criteria that you want to filter by. Fuzzer nameIf you only want to see crashes found while running a particular fuzzer, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found by the selected fuzzer while using the selected job type. Job typeIf you only want to see crashes found while running a particular job type, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found for the selected fuzzer while using the selected job type. Security flagYou can also choose to limit the list to crashes with security implications. By default, all crashes are shown. Last X daysBy default, the crash statistics page only shows crashes from the previous day. If you want to search farther back in time, you can change this here. Defining job typesClusterFuzz allows job types to be defined easily. A job definition is a collection of environment variables set on the bots running a job. These define which binary to run, how many test cases to generate, and much more. Defining a job type for an asanified chrome targetSometimes, managing a custom binary is not ideal. This usually happens if you are testing an area of the code that changes frequently enough that you do not want to manage a custom binary, but also cannot pass test cases to chrome directly. This most commonly comes up when working with IPC handlers. Often, fuzzer test cases cannot be passed to Chrome directly. Whenever possible, a build target should be added so that ClusterFuzz can pick up changes automatically, and so that it can do additional analysis such as regression or fixed testing.
Job types for builds from this location should include the following definitions: RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-release-([0-9]+).zip SYM_RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-symbolized-linux-release-([0-9]+).zip SYM_DEBUG_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-debug-([0-9]+).zip REVISION_PATTERN = .*-([0-9]+).zip STABLE_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-stable-([0-9.]+).zip BETA_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-beta-([0-9.]+).zip VERSION_PATTERN = .*-([0-9.]+).zip These tell ClusterFuzz where to pull its builds from. You will then need to define an APP_NAME, which is the name of the binary within the archive that should be executed, and APP_ARGS, which represents how the command line for running your application should be formed. APP_NAME = my_target APP_ARGS = %TESTCASE% The previous example is the simplest case possible and would launch a command of the form “/path/to/my_target fuzz-mytestcase-1”. You can add as many command line arguments as you need. If your test case is passed as a variable, APP_ARGS can be defined as something like the following: APP_ARGS = --flag-1 --testcase-file=%TESTCASE% Defining a job type with a custom binaryUsing a custom binary is much simpler than working with an asanified chrome target, but has the disadvantage of needing to be manually updated. When defining the job type, simply upload an archive along with the job definition. This will be the custom binary associated with the job. The name of the binary itself should be specified by the APP_NAME variable, and the format for the command should be specified by the APP_ARGS variable. “CUSTOM_BINARY = True” must also be defined for job types that use custom binaries, but this will be added automatically. A sample job definition that uses a custom binary follows: APP_NAME = chrome APP_ARGS = --snipped-some-arguments --ppapi-flash-path=%APP_DIR%/libpepflashplayer.so --user-data-dir=%USER_PROFILE_DIR% %TESTCASE% USER_PROFILE_ARG = --user-data-dir NET_LOG_ARG = --log-net-log WINDOW_ARG = --window-size=$WIDTH,$HEIGHT --window-position=$LEFT,$TOP MAX_FUZZ_THREADS = 4 TEST_TIMEOUT = 18 MULTI_PROCESS = True CUSTOM_BINARY = True Advanced optionsVariables in job definitionsA job definition is made up of a set of variables that will be defined as environment variables when ClusterFuzz runs. Some commonly used ones are included below, but this is not intended to be a complete list.
Command format stringsYou may have noticed that APP_ARGS supports some strings with special meanings, such as %TESTCASE%. A full list of these is included in the following table.
|
|
|