This page has details to help Chromium perf sheriffs. For more information, see Tree Sheriffs and read the theses on Sheriffing.
Keep performance regressions from making it out into the wild! In particular, triage all of the Chromium Perf alerts and keep the Chromium Perf waterfall green.
The Chromium Perf waterfall runs many different tests against the latest Chrome to measure Chrome’s performance. This data is automatically analyzed by GASP and will report anomalies via email to the current perf-sheriff, perfbot-gasper@ and on the perf dashboard. These anomalies likely correspond to a performance regression.
You'll be last line of defense in upholding Chrome's core principle of speed.
"There is usually a way to add functionality without hurting page load time, startup time or other key metrics. One just has to be a bit more clever at times." -Darin Fisher
What to Watch
It is also recommended to hang out in #chromium on irc.freenode.net during your shift. Most chromium developers are there, so it can be useful for pinging people about issues.
Keep the perf sheriff tracking doc up to date with all known issues.
What to Do
Keep bots green
- Purple bot? Purple indicates bot problems. Ping chrome-troopers at google dot com.
- Red or Orange bot? Red indicates first failure. Orange indicates repeat failure.
- Scan the build history for the first failure. Click it and analyze the CLs in that build for what could have caused the failure.
- File a P0 bug with the Performance and Type-Bug-Regression labels. Make sure it contains a link to the failing step build log.
- If you are able to identify the culprit change with reasonable certainty, add the author to the bug and in the meantime politely revert the CL directly with drover. No need to wait for the author to confirm in the case of a breakage.
- If you are not able to identify the culprit, try to reproduce locally and/or CC possible culprit authors on the bug. Consider also reaching out to chrome-perf-sheriffs at google dot com and/or tonyg.
- If the failure appears to be flake, file a P2 bug with the Performance and Type-Bug-Regression labels. Feel free to investigate only if you have time.
By the end of your shift, try to leave the next sheriff with a green tree!
Triage dashboard alerts
By the end of your shift, try to leave the next sheriff with an empty alerts page!
- As perf-sheriff, you are responsible for following through with the regression bugs filed during your shift. Assign yourself as the owner until you find a more appropriate owner. Be sure the bugs have the labels Performance and Type-Bug-Regression so they show up in our weekly triage meetings. To further prevent severe regressions from slipping through the cracks, consider applying milestone and release block labels to the bug.
- Click the dashboard's "Bisect" button on the alert to kick off a bisect job to pinpoint the culprit change. The dashboard will run the bisect and update the bug with the results when complete.
- When the bug gets the results, verify that they make sense. The magnitude of the regression detected by the bisect should roughly match that on the dashboard and the confidence should be high. If not, change something about the bisect and try again. For example: more iterations, a wider revision range, a different platform, or a more specific test (e.g. one page of the page_cycler or one suite of dromaeo).
- Once you find the culprit change, assign the bug to the author and ask them to revert. Chrome has a no-regression policy as specified in our core principle of speed. Because there are sometimes tradeoffs involved, or other considerations, it usually is best to let the author do this rather than doing it yourself. If the author claims the regression is expected but you have any doubts, feel free to loop in rschoen/tonyg or reach out to chrome-perf-sheriffs for more support.
Handle sizes regressions
- If sizes fails and the jump is > 100K or static initializer fails, talk to the build sheriffs in #chromium about identifying the culprit and reverting.
- If sizes fails and the jump is < 100K, update the expectations.
- svn co svn://svn.chromium.org/chrome/trunk/src/tools/perf_expectations/
Edit perf_expectations.json. (Also consult make_expectations.py for more info).
Find the line corresponding to the new failure. For example, if you're looking at a failure on the perf dashboard in linux-release-64, the sizes test, and a failing expectation for 'chrome/chrome', the line you need to update will have the key 'linux-release-64/sizes/chrome/chrome'
Select a new reva and revb. Include the last 50 revisions and set a tolerance of 5% The new range should have a 5% increase over the last result (no need to change the values, the 'make_expectations.py' script will take care of that below).
For that test result, update reva and revb to match the new range and save your changes to the file.
Run 'make_expectations.py', this updates perf_expectations.json with new results values and a new checksum for that line.
Open a new bug (example) with label Performance and cc all authors of CLs in the regression revision range. Assign the bug to the first or most likely author in the list and ask them all to verify if their change could have introduced the regression.
Upload a CL (example) with your change. Like the example, include in the commit message the key that is being updated along with a working persistent link to the change on the graph. Make sure the presubmit tests run, since these tests check for common syntax errors. Send your CL for review to the current perf sheriffs (check the calendar if you're unsure). Include at the bottom of the description:
BUG=<your new bug>
Publish your CL on codereview.chromium.org so that it can be reviewed.
If the tree is open, land your CL then notify the sheriff(s).
- If the tree is closed, check with the sheriffs if you are free to land your change. Let them review it if they want, in addition to the TBR.