Legacy Chromium OS Commit Queue Overview

None of this applies to Parallel CQ. For that, see the Sheriff FAQ.

This page describes the Gerrit Commit Queue for Chromium OS (also known by the code name "Paladin")

The goal of the Commit Queue is to vet changes (build and test them on multiple platforms) and push them on behalf of developers.  It is responsible for not only committing changes, but also doing the Portage voodoo (uprevving ebuilds) necessary for other developers to see each others changes.

How do I submit my change to the Commit Queue?

If your change has been reviewed and marked as +2 ("Looks good to me, approved"), you can submit your change by selecting the Reply button and marking your change as both "Verified +1" and "Commit-Queue +2".

How does the Commit Queue verify changes?

We first launch trybots to sanity check your change. This step is called PreCQ (Pre-Commit Queue). The trybots will compile your change, build images, and run unit and VM (virtual machine) tests on major platforms or platforms configured for the corresponding repositories/directories. This check only includes your change and related changes, and takes about 40 minutes.

The commit queue runs approximately once every 3-4 hours, and picks up all changes that have been fully verified by PreCQ. It builds a full image (on a set of representative platforms) and runs tests on both virtual machines and real hardware. Each commit queue run takes approximately 3-4 hours, and a new run is kicked off immediately after completing the previous run.

How does the commit queue decide when to pick up a change?

The Commit Queue only picks up changes that have been approved, verified and marked ready, and only if all their dependencies are similarly approved, verified, and marked ready (see the "How do I specify the dependencies of a change?" section below).

  • If the tree is open, the Commit Queue will start a normal run and pick up all eligible changes (see How do I submit my change to the Commit Queue above).
  • If the tree is throttled, the Commit Queue will pick up Commit-Queue+2 changes if they are available, or pick up a set of changes to test. In the latter case, the changes will not be rejected even if the build fails because the tree is considered not sane (since it is throttled). After a failed run, the subsequent run will attempt with a smaller set of changes.
  • If the tree is closed or under maintenance, CQ will not pick up any changes.

Why wasn't my change picked up by Commit Queue?

If your change was picked up by the Commit Queue when the tree was throttled and the run failed, the Commit Queue would attempt to progress by selecting a smaller set of changes in the next run. As the results, your CL may be skipped for the subsequent run. The Commit Queue will continue reducing the number of CLs until a run completes successfully. You do not need to do anything because the Commit Queue will pick up your change for sure once the tree is open or it completes a successful run. If you still suspect there was some wrongdoing of the Commit Queue, please contact the build deputy to triage the issue.

When should I mark my changes as dry run ready?

The dry run ready flag (CQ+1) lets committers kick off trybots before your change is approved. Select the CQ+1 flag in Gerrit and it'll kick off a trybot. The status will be reported back to Gerrit. If you later mark the same patch as CQ+2, the trybot will already be complete, so your CL may be committed up to an hour faster.

You don't have to use the CQ+1 flag -- if you don't, the CQ will kick off trybots anyway when your CL is approved.

Will the Commit Queue respect dependent changes in Gerrit?

Yes. This means that the Commit Queue will not try a change that depends on another until both it and the change it depends on are marked as ready for commit.

How do I specify the dependencies of a change?

See the Cq-Depend documentation.

How can I launch a non-PreCQ trybot for my change?

If you want to test your change with non-PreCQ configs, see the remote trybot documentation.

What repositories are currently managed by the Commit Queue?

All projects under chromiumos/* and chromeos/* in gerrit and gerrit-int respectively are managed by the commit queue.

How do I watch what the Commit Queue is currently doing?

In order to see what the Commit Queue is doing, you can view its current work on the internal and external waterfalls.

  • The CommitQueueSync step shows what builds the current Commit Queue build is testing 
  • The CommitQueueCompletion step is responsible for submitting changes when they pass these tests

What happens when the Commit Queue rejects my change?

Depending on the scope of the test failures, the CQ may reject one, some, or all of the changes which were attempted during that run.  The ChromeOS Commit Bot will set the Commit-Queue mark on any rejected change to 0.

The ChromeOS Build Sheriffs and their deputies will attempt to attribute build and test failures to a particular root cause.  When they have annotated and finalized their remarks on a failed build, the ChromeOS CL Exonerator Bot will go through and add Commit-Queue +2 back onto the CLs which were previously rejected, such that they may be retried on the next run of the CQ.

If the Build Sheriffs suspect that the root cause was due to a particular CL, they will report Verified -1 on the suspicious CL.  Note that the threshold of suspicion is low, and that the sheriffs may not be experts on your subsystem!  If a sheriff has blamed your CL, then you have been deputized to investigate the build failure in greater detail.  It could have been your CL, a problem with a earlier commit, a different CL in the same CQ attempt, etc.  Even if your change wasn't the cause of the build failure, you are likely in a better position to investigate the cause than the sheriff was.

My change was rejected by the Commit Queue, what can I do?

Possibly nothing.  If your change was not at fault, then the process of review and exoneration will automatically retry your change until it is committed.  If it has been more than 4 hours since your change was rejected and it has not yet been exonerated, then you may mark it Commit-Queue +2 again manually after verifying that your CL did not cause the CQ failure.

When the CQ rejects your change, it adds a comment to Gerrit with additional information about why it was rejected.  There will be a link to each of the failed builders' logs with URL's that end in $(board-codename)-paladin/$(build-number).  You can also look at the commit-queue builders on the internal and external waterfalls and find the build run which has your CL in the CommitQueueSync stage.  These are the same logs other builders would have so delve accordingly.

Logfile hints:

  • master-paladin serves primarily to execute the other builders and aggregate their responses.  The most interesting logs are usually found on the slaved paladins.
  • Logs are managed by several systems.  Links into any of the following are links about the logs: Luci, Milo, Stainless, LogDog, and Goldeneye.
  • (Googler-specific) Stainless is not multiple-login aware.  If your browser session is logged into both @google.com and @chromium.org credentials, and you are presented with an authorization error (Forbidden 403), they try appending "?authuser=1" to manually select your other credentials.
  • In the builder's log, failures in BuildPackages and BuildImage correspond directly to the build_packages and build_image steps in a normal developer workflow.  Failures to compile or package a CL will show up here.
  • Failures in tests that are managed by 'autotest' will include a tree of logfiles, including logs from the device under test and logs from the system driving the test.

There are three places where the commit queue can fail to commit your change.

  1. When applying a change: If your change conflicts with another change, your change will be rejected and you will need to rebase your change locally and resolve the conflicts.
  2. When verifying a change: This is the standard failure where either we fail to build or test your change. Please read the error message and verify that the failure was not caused by your change. If your change was not at fault, you may mark it Commit-Queue+2 again manually.
  3. When submitting a change: This can happen if either you already submitted your change while the commit queue was working or someone else bypassed the commit queue and submitted a change that conflicts with your change. If the latter, follow (1).  Additionally, if a revised patchset was published to Gerrit while the CQ was evaluating it, or the change was Verified -1 prior to completing the CQ run, then the change will also be rejected.

Will the Commit Queue commit my CL when the Tree is Closed?

No. However, it will commit any CLs it is processing if the tree is Throttled.

e.g. Tree is open, CQ picks up your CL, tree goes to throttled, CQ finishes testing & merges your CL.

How can I bypass the Commit Queue?

Please do not bypass the commit queue unless it is necessary. That said, to bypass the commit queue, hit the submit patch button and select "Yes, I'm a chump".  Please consult #crosoncall prior to chumping in your change, since it may complicate ongoing failure investigations.

Can I bypass the hardware test step?

Each repository has a COMMIT-QUEUE.ini file. You can control what steps to skip using this file.

Does the Commit Queue ever pick up changes that haven't been verified by (PreCQ) trybots?

If the Commit Queue finds that there are no changes that have been verified by trybots, the commit queue will start picking up unverified changes. This should only happen either at off-hours (when few developers are submitting changes) or when the trybot waterfall is having issues.

What should I do if the Commit Queue itself is down?

If the Commit Queue is down or consistently broken, please close the tree and contact a Build Sheriff or trooper immediately (chrome-troopers [at] google.com).

More Questions

Contact: chromium-os-dev@chromium.org