The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.
For Developers‎ > ‎How-Tos‎ > ‎

GPU Bots & Pixel Wrangling

GPU Pixel Wrangling is the process of keeping various GPU bots green.  On GPU bots, tests run with real GPU, not software rasterizer like other bots.

GPU Bots

Waterfalls

The waterfalls work like any other; see the Tour of the Chromium Buildbot Waterfall for a more detailed explanation of how this is laid out. We have more subtle configurations because the GPU matters, not just the OS and release v. debug. Hence we have Windows Nvidia Release bots, and Mac Intel Debug bots, and so on.

The waterfalls we’re interested in are:

Test Suites

All the GPU bots run the following test suites:
Performance tests
  1. gpu_frame_rate_tests
  2. gpu_latency_tests
  3. gpu_throughput_tests
In-process browser tests
  1. WebGL conformance tests: src/content/browser/gpu/webgl_conformance_test.cc
  2. Crash tests: content/browser/gpu/gpu_crash_browsertest.cc
  3. Pixel tests: src/chrome/test/gpu/gpu_pixel_browsertest.cc
  4. Feature tests: src/chrome/test/gpu/gpu_feature_browsertest.cc

Wrangling

Prerequisites

  1. Ideally a wrangler should be both WebKit and Chromium committer. There's a wrangling schedule. If you're in rotation, there will be an email notifying you of the upcoming stint.
  2. Apply for access to the bots.

How to Keep the Bots Green

  1. Update WebGL Conformance Test expectations: src/chrome/test/gpu/webgl_conformance_test_expectations.txt.  See the header of the file a list of modifiers to specify a bot configuration.  We can specify OS (down to a specific version, say, XP or LION), Debug vs Release, GPU vendor (NVIDIA/AMD/INTEL), and a specific GPU device.  The key is to maintain the highest coverage: if you have to disable a test, disable it only on the specific configurations it's failing.
  2. Rebaseline pixel test reference images: the images are stored locally on each bot at /b/build/slave/$(SLAVE_NAME)/gpu_data/reference.  There are two ways to rebaseline the reference images.
    1. Log in to related bots and delete the entire local reference image folder.  In the next run the reference images will be regenerated.
    2. Log in to related bots and delete individual images from the reference folder to regenerate them on the next run. Delete the $test_$number.png, $test_$number.rev, and any old $test.png files in that directory for some value of $test and and $rev.
    3. Or, goto gpu_pixel_browsertest.cc, and for each test, there is a ref_img_revision_update variable.  Change its value to the revision number that changes the pixel test results, and any reference images generated from older revision will be blown away and a new set of ref images will be generated.
  3. All other tests, do the regular DISABLE_ / FLAKY_ / etc thing.
  4. (Rarely) Update the version of the WebGL conformance tests. See below.

Of course, before doing any of the above, check first if the redness is a regression.  If yes, revert (or if you understand what's going on and have the time, fix it directly).
Breakages can be caused by either WebKit or Chromium changes.

When Bots Misbehave (SSHing into a bot):

  1. See the Chrome Internal GPU Pixel Wrangling Instructions for information on ssh'ing in to the GPU bots.

Updating the WebGL Conformance Tests

Occasionally a bug in the WebGL conformance tests will be exposed by a WebKit roll, and the best solution is to roll forward to a new version of the WebGL conformance suite in which the bug has been fixed. In order to do this, follow the steps below. (Note: this mechanism is not ideal. There appear to be issues attempting to create a Subversion mirror of Khronos' repository. "svn patch" would be a better way to achieve this result, but requires a version of Subversion not commonly available.)

  1. Check out a clean copy of the WebGL conformance tests:
    1. mkdir khronos
    2. cd khronos
    3. svn co https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl
  2. Check out the top of tree copy of these tests from the Chrome tree:
    1. mkdir chrome-deps
    2. cd chrome-deps
    3. gclient config svn://svn.chromium.org/chrome/trunk/deps/third_party/webgl
    4. gclient sync
  3. Delete the entire "tests" directory out of the deps/ copy:
    1. cd webgl/sdk
    2. svn del tests
  4. Upload and commit a CL (TBR'd) for this deletion. Make sure you indicate in your commit message the revision in the Khronos repository to which you are syncing!
    1. gcl change deleteeverything
    2. gcl upload deleteeverything
    3. gcl commit deleteeverything
  5. Delete all of the Subversion metadata out of the "tests" directory in your Khronos webgl checkout:
    1. cd .../khronos/webgl/tests
    2. find . -name \*.svn -exec rm -rf {} \;
  6. Copy the entire Khronos webgl checkout into your deps directory:
    1. cd .../chrome-deps/webgl/sdk
    2. cp -r .../khronos/webgl/tests .
  7. Add it to the repository and commit. Make sure you indicate in your commit message the revision in the Khronos repository to which you are syncing!
    1. svn add tests
    2. cd ..
    3. gcl create addeverything
    4. gcl upload addeverything
    5. gcl commit addeverything
  8. In your Chrome checkout, update src/DEPS to point src/third_party/webgl_conformance to the version you just committed. Run gclient sync.
  9. Regenerate the WebGL conformance test list:
    1. cd .../chrome/src/chrome/test/gpu
    2. ./generate_webgl_conformance_test_list.py
  10. Build and run the gpu_tests target in your Chrome checkout to verify the update of the conformance suite was successful.
  11. Upload and try your DEPS roll:
    1. gcl change updatetests
    2. gcl upload updatetests
    3. gcl try updatetests -b win_gpu,linux_gpu,mac_gpu
  12. Test failures may be expected due to the introduction of new tests. Follow the instructions above titled "Update WebGL conformance test expectations".
  13. Once the try job is successful, commit the DEPS roll along with the update of the autogenerated source files.

Performance Test Graphs

Locating Performance Test Graphs

The performance tests on the GPU bots each have an associated graph view. To look at this graph view, find the [results] link under the appropriate test run in the column for the configuration you want (e.g. the ‘gpu_frame_rate_test’ test suite on the ‘Win7 Release (Intel)’ configuration on the chromium.gpu.fyi waterfall). So to find a graph, you go to the right waterfall, find the right configuration, scroll down until you see the test suite you want, and click on [results].

Interpreting Performance Test Graphs

On all graphs clicking on a point on the graph will show you what Chrome revision was being tested, and all the changes that went into that revision. Unfortunately all graphs (even those from the chromium.webkit waterfall) only show Chromium changes, not WebKit changes.

gpu_frame_rate_tests

A number of graphics demos. Tests are run with and without vsync on on both the current revision and a reference build.

All result numbers are in ms per frame (lower is better). The graph has four lines for the four possible configurations:
  • interval_gpu : the test run with default flags (?)
  • interval_gpu_ref : same, but for the reference build
  • interval_gpu_novsync : test run with default flags (?)
  • interval_gpu_novsync_ref : same, but for the reference build

gpu_latency_tests

A couple of input latency tests.

All measurements are reported in units of latency frame count (fewer frames of latency is better). Right now comparisons need to be made between the “software” test and the “webgl” test to see how much worse GPU latency is vs. software. The graphs each have lines for various input scenarios:

  • simple: base test setup; just a page doing simple software or WebGL rendering during RAF.
  • id: input dirty (touches the DOM during javascript input handler).
  • ih: input heavy (spin waits in input handler for 2x normal frame == ~30ms).
  • ph: paint heavy (in GPU mode, this does a bunch of glClears; in software mode, this does a full repaint -- not really a fair comparison, so take it with a grain of salt).
  • rh: raf heavy (spin waits for 2x normal frame time during RAF).

gpu_throughput_tests

A set of contrived tests comparing canvas performance on the GPU and in software.

These tests all run with vsync off, so requestAnimationFrame will run as fast as possible.

All result numbers are in ms per frame (lower is better). The graph has two lines:
  • gpu: results with accelerated Canvas2D enabled
  • software: results with accelerated Canvas2D disabled

Extending the GPU Pixel Wrangling Rotation

    See the Chrome Internal GPU Pixel Wrangling Instructions for information on extending the rotation.

Comments