The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.
For Developers‎ > ‎

bisect-builds.py

Have you ever worked on a regression bug like this: "In chromium 6.0.401.1, all renderers idle at 1.5 %cpu. In chromium 6.0.201.0, they idled at 0.0 %cpu"?

A good way to attack bugs like this – where it's unclear what change could have caused the regression, but where you have a reliable repro – is to use tools/bisect-builds.py. The script automates downloading builds of Chrome across a regression range, conducting a binary search for the problematic change. If you don't have a chromium checkout, you can download it here.

Run it like this:

  python tools/bisect-builds.py -a platform -g good-revision -b bad-revision -- flags-for-chrome

For example,

  python tools/bisect-builds.py -a mac -g 3894 -b 4000 -- --no-first-run --user-data-dir=/tmp

Valid archive types (the -a parameter) are mac, win, linux, and linux64.

You can also use the -p option to specify a profile. If no -p or --user-data-dir option is specified, a new profile will be created in a temporary directory each time you are asked to try a build.

To find the revision number to use for the "good" revision, just manually download some old revision from the continuous build (go to build.chromium.org and then click "continuous" in the upper left corner), verify it doesn't have the problem, and use that.

The script will download a build in the revision range and execute it. You must then manually check if the bug still repros. Quit Chromium, and the script will ask you if the bug reproduced or not. It will use your answer to drive a binary search, and after just a few steps it will tell you "this regression happend somewhere between revisions 1234 and 1334". From that list, it's usually easy to spot the offending CL. If you're adding the range as a comment to a bug, please always paste the output from bisect-builds.py, as this includes links to the chromium and webkit changes in the regression range.

View SVN changes in revision range with this Useful URL (replacing SUCCESS_REV and FAILURE_REV with the range start and end):

Getting an initial revision range

If you have two Chrome binaries, one which doesn't work, one which does, you can check the chrome://version page for the revision that it was built at (look for the "(Official Build NNNNN)" text). You can also infer the revision number from the version number. The third set of numbers in a version number (e.g. the 835 in 14.0.835.202) is the branch number. You can look up http://src.chromium.org/viewvc/chrome/branches/<branch_number>/src/ and see find messages of the form "Branching for <branch_numebr> @<revision_number>".

Use an elevated command prompt for use on Windows

The script currently requires an elevated command prompt to extract the build on Windows.
You can run cmd or cygwin as administrator to work around the silent failure. Track this issue at crbug.com/98739.

Argh, the offending CL is a Blink roll!

This can be annoying, especially if it's a big roll. You can use http://src.chromium.org/viewvc/blink?revision=87019&view=revision or http://build.chromium.org/f/chromium/perf/dashboard/ui/changelog_blink.html?url=%2Ftrunk&range=150115%3A150138&mode=html to manually look at the changes in the roll, maybe something looks obvious. If that doesn't help, a somewhat time-consuming but reliable method is to sync your repo to that roll and then to manually bisect the Blink revisions (check out middle revision, build, check if problem repros, repeat).

Here is an example of doing that using the git flow:
  • First make sure that you have checked out the correct version in the Chrome repo:
    git checkout $(git svn find-rev r<chrome_revision_number>).
  • Sync to that version so Chrome builds:
    gclient sync -r src@HEAD
  • Compile ninja -C out/Release chrome (these 3 steps can be repeated for a normal manual Chrome bisect).
  • cd third_party/WebKit
  • Find the Blink revision you want to test and copy the hash from the top.
    git log origin/master --grep @<blink_revision_number> 
  • Then check that version out: git checkout <blink_hash> (or if you want to do it in a single command:
    git checkout $(git log origin/master --grep @<blink_revision_number> | head -n 1 | cut -d" " -f 2)
  • Make sure everything is set up correctly.
    build/gyp_chromium 
  • Compile and run:
    ninja -C out/Release chrome
    out/Release/chrome
  • Repeat.

If the problem repros in WebKit nightlies (not that likely now that we're using Blink), you can use Tools/Scripts/bisect-builds from the webkit repo. Or:
Comments