For Developers‎ > ‎

Tree Sheriffs

For Chromium sheriffing, we are migrating the wiki into markdown in the repo - see Chromium Sheriffing for the latest information on Chromium Sheriffing.

For Chromium OS Sheriffing see Sheriff FAQ: Chromium OS

I'm a new sheriff, now what?

  1. Read the tour of the waterfall to understand what you are doing.
  2. Read this page
  3. Read the in-tree sheriffing guide doc
  4. Join #sheriffing on chromium.slack.com
  5. Read the relevant page for your sheriffing type:
    1. Chromium sheriff
    2. Memory sheriff
    3. Chromium OS sheriff
    4. NaCl sheriff
    5. GPU pixel wrangler
    6. Chromium Perf sheriff
We also have non-tree sheriff rotations: Stability sheriff, Security sheriff, and Blink bug labeling

What is a sheriff?

  • Closes and opens the tree
  • Tracks down people responsible for breakage
  • Backs out broken changes
  • When idle, the sheriff:
    • improves the tools,
    • updates the doc,
    • fix flaky tests,
    • remove reliability signatures
Sheriffs should have a strong bias towards actions that keep the tree green and then open. For main waterfall bots that are on the Commit Queue, if a simple revert can fix the problem the sheriff should revert first and ask questions later. For bots that are not covered by the Commit Queue and if the author is online, it's fine to ask them to fix asynchronously (since it shouldn't be blocking anyone, and it's not the author's fault as the change landed through the CQ).

What is Sheriff-o-Matic?

Sheriff-o-Matic (sheriff-o-matic.appspot.com) is a tool to automate the work of sheriffing. It shows you just the failures and it automatically intersects regression ranges across the bots for you. Currently it is used for the Chromium tree. If you're interested in making it work for other trees, see: crbug.com/409693.

What is Findit & Flake Portal?

Findit is the culprit finder and auto-revert service for compile/test failures and flaky failures. Once a failure/flake occurs on CI/CQ, Findit is automatically triggered to identify and auto-revert the bad commits. Results are surfaced to Sheriff-o-Matic too.

Flake Portal is the entry point for flakiness. Here describes its functionalities: detection & ranking flaky tests by negative impact, analysis to find culprit, verification of whether a test is still flaky, etc.

All feedback are welcome to findit@chromium.org or via filing a bug here.

What is a trooper?

Troopers know more about maintaining the build infrastructure. They're the people to look for when the bots need an OS update, a machine goes offline, checkouts are failing repeatedly, and so on.
  • Chromium troopers:
  • Chromium OS troopers:
    • Same list as Chromium troopers (see above)

What is a gardener?

Gardeners are watchers of particular component interactions. They generally watch a component's release or development and move the version included forward when it is compatible. This is called "rolling DEPS", and consists of committing a CL that changes the (Subversion) revision number in some DEPS file, particularly the root DEPS, and then seeing if anything breaks. In some cases the DEPS roll also needs to have a landmine to force a full rebuild (clobber).

Of particular interest to the Chromium projects are the gardeners who watch the interaction between Skia and Chromium, and those who watch the interaction of Chromium and ChromiumOS.

Our 7 out of 8 goal

  • We have a goal of keeping the tree open 7 hours out of every 8. To achieve this goal, sheriffs need to resolve issues as they arise.
  • If the tree is closed automatically, the sheriff needs to change the status within a few minutes to let other people know that someone is looking at the failures.
  • If the sheriff can keep track of the failures on the tree, and those failures are not catastrophic (e.g. compile failing or thousands of test failing), then the tree should stay open.
  • Want to know how you did? Check the status viewer for ChromiumChromium OS or NaCl. (7/8 = 87.5%)

Common tasks

Closing or opening the tree

The tree status can be closed or open. These status levels control the activity of the commit queue. If the tree is open, the commit queue runs as normal.
  1. Go to chromium-status.appspot.com (Chromium) or chromiumos-status.appspot.com (Chromium OS).
  2. Change the status.
    • To close the tree, include the word "closed" in the status.
    • To open the tree, include the word "open" in the status.

Effectively communicating tree closure

Annotate the tree status with information about what is known about the status of build failures. For example, automatic closure messages such as...

Tree is closed (Automatic: "compile" on "Mac Builder" from 12345: foo@chromium.org)

... should be changed to:

Tree is closed (compile -> johnd)

... to indicate that committer 'johnd' has been notified of the problem and is looking into it. Once a fix has been checked in, sheriffs often use status:

Tree is closed (cycling green)

... to indicate that a fix/revert has been checked in and the tree will likely be opened soon. Alternatively, if the sheriffs decided to revert first and ask questions later, then the tree status should be changed to:

Tree is open (reverted r54321)

Effectively communicating tree repairs

If the tree has been closed for an extended time, particularly if the breakage covered more than one working timezone (US Pacific, US Eastern, Europe, Asia), it is considered best practice to communicate what was needed to fix the breakage. That way the next sheriff knows what's been happening, and people in other timezones know what to do next time it breaks the same way.

If the fix was simple, it can be listed in the tree-open status message, such as...

Tree open (rebooted Chromium (dbg) to clear tempfiles)

Tree open (svn server came back)

Tree open (reverted r54321)

If a more detailed fix was needed, send email to the chromium-dev mailing list explaining what happened. It's a good idea to CC the current and upcoming sheriffs too.

Tips and tricks

It's clobberin' time!

Sometimes you just need to clobber (i.e. force a full, clean rebuild of) some class of bots (win, mac+ninja, linux asan using make, etc.). You can do this by landing a landmine change. Docs are here: Chromium Clobber Landmines.

Note: if a specific CL is causing bots to break unless they are clobbered, that CL should be reverted first and fixed to avoid this.

Use chromium extensions

See Useful extensions for chromium developers for more information.

Temporarily mark a builder as experimental

If a specific builder is affected by infrastructure problems, or something else that causes the builder to consistently fail, consider using the tree status to mark the builder as experimental while investigating. By marking it as experimental, the CQ will ignore failures from that builder and will not wait for it to complete, but it will still kick off that builder each CQ run.

File a bug for the issue, and set the tree status like this:

Tree is open (EXPERIMENTAL=gale-paladin crbug.com/740411)

You can provide a comma-separated list of builders after EXPERIMENTAL=.

PFQ Builds

Documentation is here in the Pre Flight Queue documentation

Sheriff schedule

Build sheriff calendar (authoritative)

The authoritative list is on Google Calendar. Here's how to add the sheriff calendar to yours:
  1. Sign in to Google Calendar.
  2. Where it says "Other calendars - Add a friend's calendar" add the address for the calendar you want:
If you just want to see the calendars without adding them to your calendar, just follow these links:

To see who the sheriff is, click an event and look at the guest list. (Yes, it would be nice if it showed the people in the event title, but then there's the issue of the event title and the guest list getting out of sync -- no easy answer.)  To find when a specific person is going to be sheriff, use google calendar's advanced search box (click the down-triangle in the main search box), select the appropriate sheriff calendar, and type the person's username into the "Who" box.

The script/process that updates the calendars can be found in svn://svn.chromium.org/chrome-internal/trunk/tools/build/scripts/tools/sheriff.

Find out who is currently the sheriff

There are a couple ways to see who the current sheriff is:
  1. Waterfall console: Listed under "oncalls" at the top left of the page.
  2. RotaNG page: This is the tool that manages the rotations.

How to swap

Schedule conflicts happen. You may need to swap your assigned rotation dates. A good approach is to pull up the sheriff calendar and reach out to individuals (in person or by hangouts) with rotations with nearby dates to yours -- they're often more willing to swap. It is not ok to only send an email to a mailing list and skip your shift because nobody replied, make a bigger effort if necessary.

To swap shifts with someone, add them to the rotation so that the builders and other tools display the proper people as sheriffs. Instructions for swapping in rota-ng are here.

Out of office

You can mark yourself as out-of-office using RotaNG in order to ensure that you're not scheduled to sheriff while you're on vacation or on leave. Note that this only works if you mark yourself as OOO before you're scheduled. If you've already been scheduled to sheriff, you should try to swap with someone. If swapping is impossible, or if you're leaving the Chromium project permanently, file a bug here.
  • Every committer is empowered and encouraged to do any of those things when needed, but the sheriff has overall responsibility in case somebody else is away or not paying attention.
  • The sheriffs receive gatekeeper emails when the tree is being closed automatically. Please take action on these as soon as you can.
  • Everyone helps! Committers must request to be signed up to be a sheriff. If you're a new committer, file a bug here asking to become a sheriff, or ping g.co/bugatrooper. You can find your time as sheriff at the "Upcoming sheriffs" list at the end of the Sheriff details pages (e.g., for Chromium). If you need to change the schedule (you're out sick or on vacation), it's your responsibility to find a replacement for your time slot. (See also the "Out of office" section, above.)