Goal

Keep performance regressions from making it out into the wild!  Also, triage all of the Chromium Perf alerts and keep the Chromium Perf waterfall green.

The Chromium Perf waterfall runs many different tests against the latest Chrome to measure Chrome’s performance.  This data is automatically analyzed by GASP and will report anomalies via email to the current perf-sheriff and perfbot-gasper@ and also on the new perf dashboard.  These anomalies likely correspond to a performance regression.

"There is usually a way to add functionality without hurting page load time, startup time or other key metrics.  One just has to be a bit more clever at times." --Darin

What to Watch

• Perf dashboard - if you are the sheriff on duty, you will also receive corresponding email alerts from gasper-alerts@

• Chromium Perf Failures-only waterfall (live doc for tracking failures and flakiness)

• Chromium Linux sizes steps

• Chromium Webkit perf bots (just make sure these are online and running tests)

What to Do

• Waterfall down/inaccessible or is a bot offline?  Ping a trooper or file a ticket to have it be rebooted.

• Bots too far behind?  Either ask sheriffs to throttle the tree or file a ticket to get more bots.

• Test red because it failed?  File a bug and revert the CL that broke the test. Check to make sure the tracking doc is up to date.

• Triage performance alerts from GASP and revert CLs that cause regressions.

Triage anomalies from GASP

• Visit the alerts dashboard page (sorted by default by most recent alerts) and make sure that the Perf Sheriff rotation is selected.

• Click "sign in" in the top right corner and use your google.com account.

• Triage every alert listed, first by viewing each graph.

• If the alert is a real regression, file a bug and track down the right owner (see detailed instructions below).

• If the alert is invalid (i.e. GASP alerting on noise), mark alert as invalid or assign bug id -1.

• Check the _ref trace as well (shown on graph by default) to see if it was a change in the machine.  If both the trace of interest and the _ref trace move equally, something changed on the bot and it is not a regression in Chrome

• By the end of your shift, the alerts dashboard page should have zero untriaged alerts.

• As perf-sheriff, you are responsible for following through with the regression bugs filed during your shift.  Assign yourself as the owner until you find a more appropriate owner.

• If you aren't sure what to do, email chrome-perf-sheriffs@ and cc chrome-perf@ for backup.

For each regression, file a bug

• You can file a bug directly from the alerts page or the graph page.

• Label the bug with a milestone blocker for the next stable version of Chrome that could potentially include the regression if not fixed.

• Assign yourself as the owner until you find a more appropriate owner.

• For the fastest, best results it helps to identify the suspect CL(s) when filing the bug.  Sometimes this is obvious from looking at the change list (click on a point on the graph to see the change list below).  Most of the time it is not.

• Use the Performance Regression Bisection Script to narrow down the CL(s) in question - this script can currently be run on Linux, Mac and Windows trybots.

• Check the other perf bots to narrow down the possible revision range, namely the Blink canary bots.  Note that they always run ToT Blink, so if the regression is due to Blink, it might occur at a different Chromium CL.

• If you've narrowed down a regression to a Blink roll, try to narrow down the Blink regression range by checking the Blink canary bots or using the bisection script, which can automatically drill down into Blink CLs. Don't assign it to the Blink gardener.

• If you cannot narrow down the offending CL, you should speculatively revert suspicious CLs

• After filing a bug from the graph view, make sure to enter the bug id and click submit to associate the bug with the alert

Reverting CLs

By default, we operate on a no regressions policy.  If you can identify a CL that causes performance regressions, revert the CL.

Known corner cases:

- If sizes fails, just update the expectations. Take a look at the graph and make sure there aren't any jumps of more than 100KB. If there are, file a bug first.

- If a test only fails on one bot by a few percent, ignore it and update expectations if necessary. This is especially true for older bots, like linux-lowmem and win-xp. You may be able to narrow down bot-specific flakiness by checking the WebKit canary bots too, since they run a similar config.

Join Us

If you'd like to help keep the Chromium Perf waterfall green, send email to cmp@chromium.org. The rotation lasts 3 days every 3-ish weeks. The more that sign up, the less often the rotation will happen.

Calendar