Chrome Pre Flight Queue (PFQ) FAQ

Contents

  1. 1 DEPRECATED: this is now replaced by automatic Chrome uprreving on CQ. (nednguyen - 11/19/2019)
  2. 2 Some of this information may be out of date (skau - 07/12/2017)
  3. 3 Additional information can be found here.
  4. 4 The Pre-Flight Queues
    1. 4.1 Chrome PFQ
      1. 4.1.1 PFQs for Nightly Chrome releases (examples: x86 chrome PFQ, arm_chrome_PFQ, amd64_chrome_PFQ) Sheriffs should watch this build.
      2. 4.1.2 PFQs for ToT Chrome source (examples: chromium.ChromiumOS (x86), chromium.ChromiumOS (tegra2))Sheriffs must watch this build.
  5. 5 FAQ about this FAQ
    1. 5.1 What if a developer lands a change that is dependent on a too-new version of Chromium?
  6. 6 Who to talk to for PFQ-specific questions

DEPRECATED: this is now replaced by automatic Chrome uprreving on CQ. (nednguyen - 11/19/2019)

Some of this information may be out of date (skau - 07/12/2017)

Additional information can be found here.

The Pre-Flight Queues

This document is intended to describe our *-pre flight queue buildbots to sheriffs and Chromium OS developers.

The pre flight queue works in conjunction with the Commit Queue (paladin) builders.  Its goal is to quickly integrate closely-related packages (Chrome) without breaking the Chromium OS tree.

There is currently only one flavor of PFQ: the Chrome PFQ.  The Chrome PFQs vet various versions of the Chromium browser for use on Chromium OS. They ensure that we're using the newest build of Chromium possible while at the same time making sure we're not breaking Chromium OS developers by pulling in a version of the browser that doesn't function "correctly".

Chrome PFQ

There are two groups of Chrome PFQs: those that test nightly Chrome releases, and those that test ToT Chrome. The PFQs for nightly Chrome releases vet the release, and then update the ebuild. In contrast, the PFQs for ToT Chrome source are "informational"-only. They test the ToT source, but do not publish new stable ebuilds.

These two groups of Chrome PFQs are described in more detail below.

PFQs for Nightly Chrome releases (examples: x86 chrome PFQ, arm_chrome_PFQ, amd64_chrome_PFQ) Sheriffs should watch this build.

These builders vet and bless versions of chrome for inclusion into the Chromium OS builds.  The browser issues a new "release" (at least) every night.  This is not actually a release-caliber build or anything, but simply a spec that gives a pinned revision of the browser source, and pinned revisions of all the third-party code upon which Chromium relies.  Whenever a new Chrome release is published, the Chrome PFQ will attempt to vet this release and push a corresponding stable ebuild.  For example, when Chromium releases 10.0.627.0, the Chome PFQ creates a chromeos-chrome-10.0.627.0_rc1 ebuild (using chromeos-chrome-9999.ebuild as a template), vets it, marks it as stable, and then pushes it into the chromiumos-overlay git repo.  Ebuilds for these “latest releases” are distinguishable by this _rcX suffix.

In general, developers always use the latest _rc when building Chromium OS.  Therefore, if the latest release fails to build continuously, sheriffs should be very careful at investigating why this is happening.  If it is a Chromium bug, sheriffs should hand the issue off to the Chrome on ChromeOS gardener (listed in the chromium build console), who should do the following:
  1. determine the version of Chromium that the PFQ is building,
  2. determine the previous version of Chromium,
    • Go to the BuildTarget stage of the PFQ, and search for chromeos-chrome.  Both version strings should be in the first line you find.
  3. use https://omahaproxy.appspot.com/revision to resolve these version strings to Chromium revision numbers
  4. Consult the ToT PFQs to look for the failure in that revision range
  5. Get the ToT PFQ passing (eg. by finding and reverting the offending CL, or filing an appropriate pri-0 crbug)
  6. If there hasn't been an uprev in awhile or there is an urgent need to uprev Chrome, you can contact a Chrome TPM to discuss publishing a new release with the fix (potentially from a branch). Otherwise we normally just wait for the next scheduled release (eg. the next night) to try another uprev.
If the issue does not appear to be Chromium-related, then the Chromium OS tree should be closed until it is resolved.

There is one exception.  It is possible that a Chromium OS developer makes a change to chromeos-chrome-9999.ebuild that is not backwards compatible.  For example, adding a new build target that does not exist in yesterday’s browser source.  This would cause the Chrome ToT PFQs to stay green while the Chrome PFQs begin to fail.  This is OK; we will go green once Chromium cuts a new build.

PFQs for ToT Chrome source (examples: chromium.ChromiumOS (x86), chromium.ChromiumOS (tegra2))
Sheriffs must watch this build.

This PFQ is probably the simplest to understand of the Chrome family.  It checks out ToT browser source, builds it and runs the CrOS integration tests.  It does create and push ebuilds for the “versions” it vets, but these ebuilds are not marked as stable. They are distinguishable by their _alpha suffix.

If these PFQs are red, it means that ToT Chromium either does not build, or does not pass our integration tests.  Start by finding the faulty Chromium revision number by searching for "SVN Revision" in the build log.  If there appears to be a Chromium build failure, check the Chromium waterfall to see if it was a failure across multiple platforms, or if the revision has already been reverted or fixed.  If any of these things are true, no further action is required.  The issue is likely already fixed or under investigation.

https://crbug.com/201415 tracks making finding the SVN revision easier for sheriffs.

FAQ about this FAQ

What if a developer lands a change that is dependent on a too-new version of Chromium?

The developer should really not have done this.  As sheriff, your choices are: 
  1. Revert the change until a Chrome "release" is vetted that includes the change, or
  2. Ask Chrome-infra to push a new buildspec file with the corresponding change in it.
Strongly prefer the first option.  If there is some extremely compelling case, then you can do the latter.  Once the buildspec is published, the Chrome PFQs will kick off automatically and attempt to vet and bless the new version of the browser.
For more information, Chromium buildspec files are stored here http://src.chromium.org/viewvc/chrome/releases/?sortby=date#dirlist

Who to talk to for PFQ-specific questions

Comments