Chromium uses buildbot to run continuous builds and tests. The most useful is the Chromium Continuous master. From the top section you'll see other masters linked. Each can display a waterfall and console view. The Chromium Buildbot waterfall shows the status of many tests. That waterfall shows a lot of information that can be hard to understand at first, so let's take a quick tour.
The buildbot master process is watching the Subversion and git repository, telling slaves to start building and testing new revisions, and serves the "waterfall" page that shows all the results. (For network reasons, Chromium actually has a separate machine showing the waterfall, delayed by up to two minutes. Usually this doesn't matter, but every once in a while it can lead to confusion.). Every time a new revision is discovered, the master triggers builders as each serve a different purpose. "Testing" builders are triggered when a "building" builder doing compilation archives its build. For example, all "XP Tests (dbg)(N)" and "Vista Tests (dbg)(N)" are triggered when "Chromium Builder (dbg)" is done archiving its build.
waterfall page. For now, skip over the header (green, red, yellow, etc.) and the box at the top with lots of links and horizontal stripes (we'll come back to them later), and look down at the row of grey boxes with "changes" at the left. Those boxes show one column per build slave.
The main waterfall page shows only the builders, machines that build the executables and then distribute them to numerous machines to run various kinds of tests.
"Memory" slaves run tests using ASAN to check memory correctness, "Perf" slaves are running performance tests, and "GPU" slaves run tests related to GPU usage. "Tree closers" automatically close the tree (preventing people from committing new changes) when they fail; "FYI" testers are considered less important, so failures there shouldn't close the tree.
Click on one of the platform names next to a row of colored boxes in the big grey box at the top, and then on "waterfall" at the left, to see the testing slaves working on that platform, for example Windows. The set of slaves changes pretty often, so any list we could put here would go out of date quickly, but the names are generally pretty good at describing what kinds of machines they are and what kinds of tests they're running. A "dbg" slave is running a Debug build rather than a Release build. Some machines both build and test, but "builder" machines only build the executables, then upload them to "tests" machines to run the tests.
A yellow build step is still in progress, green finished successfully, red finished with errors, orange finished with warnings, and purple had an internal Buildbot error.
If you click on the "stdio" link for a step, you can see the exact command that was run and the environment it was run in in blue, and any output it produced in black. stderr is in red.
Most of the tests are pretty straightforward, but performance test output can be complicated at times. See the Guide to Perf Test Plots for more about that.
A tester doesn't compile, so you can't clobber it. It simply extracts its build from a builder. Testers are triggered when the corresponding builder finishes.
At the start of each run (that is, at the bottom of each series of steps for a slave), there's a yellow box holding the build number. Clicking on that build-number link shows more information about the run, including in principle the "blamelist" of changes that went into it. Every now and then, though, that list of changes is off by one. If you need to know for sure, look at the results of the "update" step to see exactly what gclient sync pulled in.
The "tree" is the sum of the various source repositories used to build the project, being Chromium, ChromiumOS, NativeClient, etc. In Chromium case, it's chrome/src/ plus everything listed in its DEPS file, and a bit more for Google Chrome like trademarked graphics. The tree can be "open", "closed" or "throttled". The normal state is open. When tests break, the tree is closed by putting the word "closed" in the tree status; PRESUBMIT.py checks the status and will block commits, and the build sheriff will act to fix the tree. When the tree is throttled, commits are only allowed with specific permission from the build sheriff, generally because the sheriff wants to make sure the tree is stable before opening it up to unlimited commits.
Banner and Box
Now back to the top of the waterfall. At the very top is a banner showing the current state of the tree. If the tree is closed because of build or test failures, it should be mentioned here. If there's an announcement about a new build process, expected downtime, or some other aspect of Chromium development, it'll generally be shown here, too.
Below that is an oval box. It has a number of handy links at the left -- try them to see where they go. It also has four rows of colored boxes, which show the pass/fail status of the last completed runs for several categories of slaves. If you click on one of those category names, you'll go to a partial waterfall view that shows only the related slaves. Hovering over a colored box shows you the name of the slave it's summarizing, and clicking on the box will go to a waterfall view with only that one slave.
Getting the Buildbot Source if you'd like to take a look or help develop it.
Since every additional configuration that closes the tree has a cost for all the developers to maintain it, there are some guidelines for new additions to the main waterfall:
If a configuration has tests that depend on code in other repositories, then it could have its own buildbot master. The team responsible for a separate configuration would sheriff that separate tree. Teams would generally find it more maintainable to have a configuration on the main Chromium tree that observes the above guidelines, even if it only contains a subset of the full code and tests. Whenever a regression is found that only reproduces on the separate team's Buildbot master, the separate team's goal should be to write a regression test that runs on the main Chromium waterfall to catch those failures in Chromium, too.
Please email email@example.com to signoff on new build configurations.
When adding new tests for existing build configs, see Adding new tests to the Main Waterfall.
It's important to use the right words so here are the official definitions: