This set of pages documents the setup and operation of the GPU bots and try servers, which verify the correctness of Chrome's graphically accelerated rendering pipeline.
The GPU bots run a different set of tests than the majority of the Chromium test machines. The GPU testers are specifically focused on tests which exercise the graphics processor, and whose results are likely to vary between graphics card vendors.
Most of the tests on the GPU bots are run via the Telemetry framework. Telemetry was originally conceived as a performance testing framework, but has proven valuable for correctness testing as well. Telemetry directs the browser to perform various operations, like page navigation and test execution, from external scripts written in Python. The GPU bots launch the full Chromium browser via Telemetry for the majority of the tests. Using the full browser to execute tests, rather than custom test harnesses like content_browsertests, has yielded several advantages: better testing code that is shipped, improved reliability, and improved performance.
A subset of the tests, called "pixel tests", grab screen snapshots of the web page in order to validate Chromium's rendering architecture end-to-end. Where necessary, GPU-specific results are maintained for these tests. Some of these tests verify just a few pixels, using handwritten code, in order to use the same validation for all brands of GPUs.
The GPU bots use the Chrome infrastructure team's new recipes framework for describing what tests to execute. Compared to the legacy buildbot infrastructure, recipes make it easy to add new steps to the bots, change the bots' configuration, and run the tests locally in the same way that they are run on the bots. There is a separate page about the GPU recipe describing how it works and how to modify it.
The Release GPU bots use the isolated testing framework to transmit the binaries and data files from the builder to the tester. (As of this writing, the Debug GPU bots use the component build, and isolates don't yet support the component build.) Running tests via isolates eliminates the Chromium src/ checkout on the test machines, achieving better utilization of the physical hardware containing GPUs.
The bots on the chromium.gpu.fyi waterfall are configured to always test top-of-tree ANGLE. This setup is done with a few lines of code in the GPU recipe module; search the code for "ANGLE".
These aspects of the bots are described in more detail below, and in linked pages.
Most Chromium developers interact with the GPU bots in two ways:
to explicitly send a try job to the GPU try servers. (It's unfortunate that the -m argument is needed; Issue 352461 tracks inferring it.) As of this writing, the GPU try servers were in the process of being added to the default set of try servers for every try job. Soon, it will not be necessary to manually send jobs to these try servers; it will happen automatically.
Alternatively, the Rietveld UI can be used to send a patch set to these try servers.
The GPU try bots explicitly support try jobs against the Chromium (src/) and Blink (src/third_party/WebKit/) trees. If you find it necessary to try patches against other sub-repositories, please file a bug with label Cr-Internals-GPU-Testing.
All of the GPU tests running on the bots can be run locally from a Chromium build. However, figuring out the exact command line to use can be a little tricky. The Release GPU bots all run their tests via isolates, and in this case, the logs from a run on the bots will contain something like
which doesn't help when trying to figure out the command line to run the test.
Many of the GPU tests are run via Telemetry. In order to run them, just build the chrome target and then invoke src/content/test/gpu/run_gpu_test.py with the appropriate argument. The tests this script can invoke is in src/content/test/gpu/gpu_tests/ . For example:
Most of the remaining tests are simple executables which can be built and run from the command line:
You can find the isolates for the various tests in src/chrome/:
The isolates contain the full or partial command line for invoking the target. The complete command line for any test can be deduced from the contents of the isolate plus the stdio output from the test's run on the bot.
The goal of the GPU bots is to avoid regressions in Chrome's rendering stack. To that end, let's add as many tests as possible that will help catch regressions in the product. If you see a crazy bug in Chrome's rendering which would be easy to catch with a pixel test running in Chrome and hard to catch in any of the other test harnesses, please, invest the time to add a test!
As of this writing it isn't as easy as desired to add a new test to one of the Telemetry based harnesses. See http://crbug.com/352807 . Let's collectively work to address that issue. It would be great to reduce the number of steps on the GPU bots, or at least to avoid significantly increasing the number of steps on the bots. The WebGL conformance tests should probably remain a separate step, but some of the smaller Telemetry based tests (context_lost_tests, memory_test, etc.) should probably be combined into a single step.
If you are adding a new test to one of the existing tests (e.g., pixel_test), all you need to do is make sure that your new test runs correctly via isolates. See the documentation from the GPU recipe on testing your own isolates for the GYP_DEFINES and authentication needed to upload isolates to the isolate server. Most likely the new test will be Telemetry based, and included in the telemetry_gpu_test_run isolate. You can then invoke it via:
Adding new pixel tests which require reference images is a slightly more complex process than adding other kinds of tests which can validate their own correctness. There are a few reasons for this.
The GPU bots are using the Chrome infrastructure team's new "recipe" infrastructure, which makes it dramatically easier than before to make configuration changes to the bots. See the documentation on the GPU bots' recipe for details on how to add new steps to the bots and modify their existing steps.