Blink‎ > ‎

Gardening Blink

Relax. Keeping track of all the Blink commits can be stressful. Sometimes the best approach is to be patient and clean up one mess at a time until the tree is sparkling green again. Blink contributors are only human, and we all make mistakes from time to time.


There are two overarching goals to gardening Blink:
  1. Increase the revision of Blink we use in Chromium without regressing any tests.
  2. Prevent or fix any Blink regressions in tip of tree Blink.
Ideally, we would increase the Blink revision in Chromium (“roll DEPS”) in small increments, more closely approximating continuous integration. That’s not always possible, but we aim to roll DEPS at least once a day to avoid falling too far behind. When things are rolling smoothly, the gardener should always have a Blink roll in the commit queue. The moment the commit queue lands a roll, queue up the next one. See closed and ongoing DEPS roll in



For goal 1, of increasing the Blink revision, fix any failing canary Chromium test buildbots. Once all these bots are green you can roll the Blink revision. You do NOT need to wait for the layout test bots to be green. But use your judgement. If there are hundreds of layout test failures or there are failures you think are likely to cause significant problems in the next Chrome Canary release, then fix those before doing the Blink roll.

For goal 2, fix any failing canary LayoutTest buildbots and failing deps LayoutTests bots using the tools listed below. Getting no failures in sheriff-o-matic is usually (not always!) sufficient to achieve this.

LayoutTests can have different expected results on different platforms. To avoiding having to store a complete set of results for each platform, most platforms "fall back" to the results used by other platforms if they don't have specific results. Here's a diagram of the expected results fallback graph.

The ASAN bot is slightly unusual; generally speaking we only care about memory failures on that bot (currently we are actually looking at the test results, but that's a bug). You can suppress ASAN-specific failures using the LayoutTests/ASANExpectations file.

The Oilpan bots are currently not tree closers and gardeners can ignore them. The Oilpan team is responsible for them (though they won't object if you have the time to help keep them running).

Gardeners should watch the Leak Bots.
Blink gardeners don't have to worry about the GPU bots (the bots whose names start with "GPU").  GPU wranglers will take care of these bots.


Generally speaking, Blink developers are not supposed to land changes that knowingly break the bots (and the try jobs and the commit queue are supposed to catch failures ahead of time). However, sometimes things slip through ...


Developers are supposed to know ahead of time which changes will require new baselines, and mark the affected tests as either "NeedsRebaseline" or "NeedsManualRebaseline" in LayoutTests/TestExpectations as part of the change that lands. 

A bot called the Auto-Rebaseline-Bot (or rebaseline-o-maticperiodically runs the "webkit-patch rebaseline-o-matic" command to update a checkout, look for new entries, run "webkit-patch rebaseline-expectations" to pull down new baselines when they are available, and then automatically land CLs with the new baselines.

So if you see TestExpectations entries for "NeedsRebaseline", you can ignore them. If you don't see a bot periodically removing these lines and landing baselines, bug someone via the list (or blink-dev if you still don't get a response).

If you see entries with "NeedsManualRebaseline", this is basically equivalent to "Failure" but indicates that someone (the developer who made the change) needs to manually review and update the baselines. You should also ignore these entries, or nag the people that added them to clean up after themselves if you're bored.


Sheriff-O-Matic is a tool that watches the Canary buildbots and clusters test failures with the SVN revisions that might have caused the failures. The tool also lets you examine the failures.

Rolling back patches

To roll back patches, you can use either git revert or drover. You can also use "Revert Patchset" on the Rietveld issue.

Flakiness dashboard

The flakiness dashboard is a tool for understanding a test’s behavior over time. Originally designed for managing flaky tests, the dashboard shows a timeline view of the test’s behavior over time. The tool may be overwhelming at first, but the documentation should help.

Contacting patch authors

Use either #blink on or comment on the corresponding Rietveld issue to contact the author. It is patch author's responsibility to reply promptly to your query.

Keeping track of ongoing issues

Fixing some problems are out of your control. You did your job filing the bug, and now it is being fixed: a bot is being brought back to life by the infrastructure team or another team is hunting down a flaky bug in browsercontentinteractive_testsorwhatevers. To keep your fellow gardeners informed of these problems, add a gardening-blink label to these bugs. This way, a gardener could always check whenever an issue pops up and comfort themselves in knowing it is under control.


Different gardeners prefer different workflows. One common workflow watches for new failures, attempts to resolve the failures as they occur, and rolls DEPS opportunistically.

Resolving Failures

Sheriff-O-Matic’s will help you watch all the canary bots for build failures and LayoutTest failures.

If a canary fails to build, fixing the compile error is the highest priority because build errors prevent us from getting test coverage:
  1. Follow Examine the link from Sheriff-O-Matic to the compile error.
  2. Determine which patch caused the regression.
  3. Contact the author of the patch and ask him to fix the failure.
  4. If the author fails to respond in reasonable time, roll out the patch.
If the failure appears to be a flaky test (e.g., because it appears only on one cycle of one bot), you can either ignore the failure or mark the test as flaky in TestExpectations. The flakiness dashboard can be helpful in guessing whether a failure is due to flakiness.

If a patch introduces a new failing test, and the author did not create baselines for various Blink configurations:
  1. Determine which patch caused the regression
  2. Contact the author of the patch and ask him to fix the failure.
  3. If the author fails to respond in reasonable time, roll out the patch.
Unfortunately, there is a window of time between when a failure occurs and when you can address the failure. Sometimes patches land during that window that cause more failures. If that happens to you, do not worry. If you calmly address each failure in turn, the process will eventually converge and you’ll have a sparkling green tree again.

Rolling Blink DEPS

The Blink AutoRollBot (ARB) script  tries to submit autogenerated rolling DEPS changes. Its current in-progress roll can be found on its Rietveld page. If it's not working, you'll have to roll manually using the roll-dep script (located in depot_tools), as described below.

Once you’ve got a Blink revision that does not contain any regressions, it’s time to incorporate that revision in Chromium.
cd src
git checkout -b blink_roll_<new_rev> origin/master
roll-dep third_party/WebKit <new_rev>
git add DEPS
git commit --no-edit
git cl upload --bypass-hooks --use-commit-queue -f [ -r <reviewer> --send-mail ]

This will do the following:
  1. Change the webkit_revision in Chromium’s DEPS to new_revision_number (the revision number can either be a git hash or an SVN revision; if the later, use a leading "r", as in "r123456").
  2. Create a changelist and add it to the commit queue.
It is a good practice to set <reviewer> to the gardener taking the next shift.

The Canary Chromium test buildbots should be an accurate predictor of your try job will fair, but running the job through the commit queue covers more configurations and can ease your peace of mind.  If you want to check whether you’ve succeeded in creating a blindingly green roll, you can look at the DEPS LayoutTest waterfall, which tests the most recent Chromium revision with the current DEPS revision of Blink.

If a Test Blocks the Roll

Blink rolls can get stopped by Chromium trybots, for instance if a V8 change breaks a downstream browser_test. If so, try to find the culprit and revert it. In emergencies you may disable the test to get the roll through, but in that case file a high-prio tracking bug with the appropriate label (for instance, Cr-Blink-WebRTC if the broken test is a WebRTC test, etc).

Handling Disasters

Sometimes a patch comes through that changes the behavior of a massive number of tests. For example, rendering tree or SVG patches can require updates to a large number of image baselines. Don’t be afraid to ask for help. We’d rather put in the extra effort to handle these cases carefully than to be surprised when a nasty regression sneaks through.

Close the tree by clicking on blink status on a build page, e.g. Canary Console View.

If all else fails, you are still stuck, and getting further and further behind (say, building up a >150-revision roll), don't hesitate to press the BIG RED BUTTON to summon the few seasoned gardeners who'd seen it all and will gladly help you out of this predicament. No, there's no actual button. That was a metaphor (I think). Just send an email to, and cc

Wrapping up the day

At the end of each day of gardening, write a summary email to As you finish your shift, look over and remove the gardening-blink label from bugs that don't have useful content for the next gardener.

Final Thoughts

Gardening Blink requires a certain amount of Zen. If you feel yourself getting frustrated, don’t be afraid to stand up from your desk and get some fresh air. If you’re calm and relaxed, you’ll make better decisions and you’ll have more pleasant interactions with other contributors.

Good luck, and thanks for helping to make Blink and Chromium better.

Gardening Schedule
See: Chrome WebKit Sheriff Calendar
Subpages (1): Triaging Gasper Alerts