Sheriff Log: Chromium OS

Sheriff: cywang
Gardener: jennyz
      • 561939: image signer stage is slow
      • 561990: Rikku: missing Manifest of chromeos-factory-board package
      • 563877: CQ failing to create valid manifest
      • 563878: shortcut broken
      Sheriff: sonnyrao, avakulenko, cywang
      Gardener: ihf
        • 561208: {Rambi-a, jecht} group machines not available in test lab for HWTests.
        • 561214: HWTestDumpJson ERROR: No JSON object could be decoded
        • 561244: bvt test got aborted but the real test job completed successfully
        • 554043: UnitTest timeouts in the CQ
        • 560915: disable Bluez flaky unit test
        Sheriff: sonnyrao, avakulenko, yoshiki
        Gardener: ihf
        • 556785: builders fail due to timeouts. build/unit test stages take over 1 hour and process is killed due to timeout.
        • Test failure in lakitu_mobbuild canary, reverted CL: 239368
        • 561036 paygen timing out on release builders

          Sheriff: ejcaruso, briannorris, wnhuang
          Gardener: afakhry
          • 554222: provision failure on falco and daisy_skate PFQs. AutoservSSHTimeout.

            2015-11-18 and 2015-11-19
            Sheriff: wnhuang, davidriley
            • 558366storm group canary build_package failed at wireless-regdb
            • 47849: update-signal-relay build failed on daisy_winter canary
            • 207003: peach_pit build_packages failed at exynos-pre-boot
            • 557578: veyron_minnie-cheets fails at chromeos-bsp-minnie-private
            • 452759: unit test timeouts on auron, rambi-a, glados
            • 516795: builds failing for exceeding 8 hour time out (auron, rambi, veyron, slippy, ivybridge)
            • 558457: ChromePFQ is all red.
            • 557449: cros_trunk is red.
            Sheriff: waihong, bfreed, dhendrix, jchuang
            • 207003: butterfly & leon build_packages failed in chromeos-touch-firmware-samus
            • 557245 (was 549044): Canary failure: The Paygen stage failed: Image signing timed out
            • 557214: build310-m2 failing to repo sync, causes CommitQueueCompletion to fail: tricky-paladin did not start
            • 557106 and 557107: Samus canary failures (HW issues)
            • 557238: Veyron_minnie recovery image signing issue ("veyron_minnie has broken appid setting")
            • 557314: Tree closer: Pre-CQ Sync stage fails to pick up CLs
              • I do not know how to find pre-cq problems in general, but these provide clues:
            • 557364: Need to recover Rialto BVT machines to get TPM into a good state.
            • 207003: peach_pit build_packages failed in chromeos-touch-firmware-pit.
            • 552648, 536689, 535928: Multiple network_VPNConnect.l2tpipsec_xauth failures

            Sheriff: kcwu, waihong, bfreed
            • 556529: Samus build_packages failed in chromeos-touch-firmware-samus
            • 25691600Network/Hardware Issue with chromeos4-devserver2, possible cause of 540587: provision Failure (Failed to perform stateful update)
            • 556671veyron canary: timeout_util_unittest failed
            Sheriff: kcwu
            • 540587: provision Failure (Failed to perform stateful update)

            2015-11-10 and 2015-11-11
            Sheriff: jrbarnette, dianders, jchuang
            • 551279 x86-zgb paladin timeout in p2p, modem-manager-next unittest (3 fails in a row again)
            • 553424: login problems, including "Malformatted response" in login_OwnershipTaken and "Timed out going through login screen" in others.
            • 554043: unittest timeouts

            2015-11-06 and 2015-11-09 
            Sheriff: waihong, rspangler, hychao, dhendrix
            Gardener: dzhioev
            • 552452 glados group canary: Failed to create HWID v3 bundle
            • 543958 veyron-b-release-group: The priority of the inactive kernel partition is less than that of the active kernel partition.
            • Attempt to resolve some spammy, mass-autofiled bugs (there seem to be a lot that have gone unnoticed for several weeks):
              • 553442: Remote power management failing for many builders
              • 553579: video_VideoDecodeAccelerator failure seen on many builders
              • 553424: login_OwnershipTaken failing on multiple builders
              • 553575: p11_replay/chaps causing network_VPNConnect.l2tpipsec_cert test to fail
              • 553548: video_VEAPerf fails on many BVT machines
              • 549910: touch_TouchscreenScroll failures on Samus and Sumo
              • 553226: cyan, celes, veyron_rialto missing from KernelVersionByBoard "expected" file in autotest
              • For next sheriff rotation: If you get bored, please look at other autofiled issues and attempt to triage and find owners for them:

            2015-11-04 and 2015-11-05
            Sheriff: dlaurie, cychiang
            Gardener: tdanderson
            • 551451 Failing BrokerFilePermission.* sandbox death tests are preventing a PFQ uprev
            • 547057 Paygen timeout
            • 547434 Paygen autotest client terminated unexpectedly
            • 548037 Paygen command execution failure
            • 551586 Paygen failed to create cache file
            • 545065 login_GuestAndActualSession_SERVER_JOB failure
            • 500094 Builder load in unittest causes some unittests to timeout
            • 542558 Mario HWtest pool health
            • 550768 bluez timeouts causing builders to fail randomly (NOT FIXED YET)
            Sheriff: bleung, deymo, cychiang
            • 550768 strago-paladin and tricky-paladin timeout while building bluez-5-r40
            • 550826 amd64-generic ASAN failed on buffet_Registration / buffer_BasicDBusAPI
            • 550840 [samus] bvt-inline test failed to remove /var/tmp/messages.autotest_start
            • 549472 [bvt-inline] security_SandboxStatus Failure on lumpy-chrome-pfq/R48-7595.0.0-rc1
            • 548535 [bvt-cq] video_ChromeRTCHWDecodeUsed Failure on tricky-chrome-pfq/R48-7589.0.0-rc1
            • 549044 The Paygen stage failed: Image signing timed out. Failure on samus-canary/7608.0.0
            • 546457 veyron_mighty interal server error on HWTest Failure on veryon_group_canary/R48-7608.0.0
            • 542558: pool: bvt, board: x86-mario in a critical state
            • 544654 [paygen_au_dev] autoupdate_EndToEndTest.paygen_au_dev_full Failure on candy-release/R48-7608.0.0
            • 551279 x86-zgb paladin timeout in unittest
            Sheriff: bleung, deymo, reveman
              • 548755 BranchUtil failure on canary master -> external and internal manifest out of sync
              • 546871 panther: git package seems corrupt while building

              Sheriff: chihchung, semenzato, shchen
              Gardener: abodenha
                • 549044 Paygen image signing timed out
                • 547541 CQ Failing in PublishUprev repeatedly
                Gardener: abodenha
                • 548693 video_ChromeRTCHWDecodeUsed test has been flaking since Oct 9
                • 548544 Compile failure on 8010 Builder #48.0.2548.0
                2015-10-27 and 2015-10-28
                Sheriff: abrestic, dbasehore
                • 548257 paygen failures which moved to ASAN only failures
                • 547057 paygen timeouts
                • 548723 autotest timeouts on ivybridge and slippy devices
                • 548755 BranchUtil failure on canary master
                • 548804 manifest broken by duplicate remote

                Gardener: stevenjb
                • 548358PDFExtensionTest.Load failing on cros_trunk

                Sheriff: johnylin, alecaberg, shawnn
                Gardener: jonross
                • 431486: Multiple PFQ failure: shill uprev build failure
                • 546865Chrome PFQ master failed while running annotated script
                • 518591: auron-release-group: HWTest flaky for test provision
                • 546871panther: git package seems corrupt while building
                • 546921lakitu-incremental: build error for chaps token_manager
                • 546630: PFQ failure: Peppy Paladin provisioning failures.
                • 415617: PFQ failure: moblab_RunSuite test failing in lab when trying to determine test platformguado_moblab paladin 
                • 545779 PFQ not upreving builds. I believe it is related to the linked bug.
                • 547055 Canary: Jecht group failed archive step
                • 547057 Canary: Rambi Paygen failure, timeout after tests pass.
                • 547116 CrOS trunk, Linux ChromiumOS, v8 roll broke a test

                2015-10-21 and 2015-10-22
                Sheriff: johnylin, alecaberg, shawnn
                Gardener: jonross
                • 546023: veryon_rialto canary failing to build rialto-services
                • 529612: lakitu_mobbuild: cloud_CloudInit fails in VMTest
                • 542558: pool: bvt, board: x86-mario in a critical state
                • 518591: [sanity] provision Failure: DownloaderException: Could not find autotest.tar in Google Storage
                • 546457veyron_mighty interal server error on HWTest
                • 544230 Bot configuration error leading to CrOS trunk browser_test failures. Have been happening since Oct 2.
                • 546567 GearMenu tests failing on CrOS trunk since introduction.
                • 546581 PFQ failure: Daisy Skate Paladin provisioning failures.
                • 546600 CrOS trunk has continuous interactive_ui_test failures for focus. Reverting suspected change.
                • 546630 PFQ failure: Peppy Paladin provisioning failures.
                • 546708 CrOS trunk compilation failure. Not sure how it made it past trybots and commit queue.
                • 545779 Ebuilds not upreving leads to incremental builder failure
                • 544751 Glados kernel build failure
                • 546639 External builders not running for days
                Sheriff: moch, zqiu, littecvr
                • 537475security_OpenFDs Failure
                • 545530lakitu: security_test_image test failure
                • 542558pool: bvt, board: x86-mario in a critical state.
                • 545588Lakitu Canary: Signer Test failure
                • 543874: [sanity] provision Failure on veyron_rialto 
                  Sheriff: moch, zqiu, littecvr
                  • 544921MTV-2081 offline due to water main burst. Lab shutdown temporarily.
                  • 545171pool: bvt, board: peach_pit in a critical state.
                  • 545172: pool: cq, board: peach_pit in a critical state.
                  • 518591: provisioning errors
                  • 543646: security_OpenFDs failing on veyron boards
                  • 543649: smaug canary failing paygen stage
                  • 543593: update_engine: buffer overflow in unittests (detected on asan bots)
                      Sheriff: vapier, grundler, bhthompson
                      • 543593: update_engine: buffer overflow in unittests (detected on asan bots)
                        • caused HW test failures on canary broadwell/braswell also.
                      • 543596: buffet_RestartWhenRegistered autotest is too flaky
                      • Tree was throttled most of the time both days. Initially due to infrastructures fixes that phobbs hadn't completely pushed.
                      • Still seeing what might be 465862 with amd64-generic-full (and others) build 15558

                      Sheriff: vapier, stevefung
                      • 541474: platform_Firewall autotest failing for various boards
                      • 543186: security_mprotect autotest failing for lakitu
                      • 518591: provisioning errors
                      • 529612: cloud_CloudInit autotest failing for lakitu
                      • 534437: SimpleTestVerify failing vmtest
                      • 543248: Linux ChromeOS Buildspec builder repeatedly failing
                      • 543593: update_engine failing unittests (buffer overflow) on asan bots
                      • 543596: buffet_RestartWhenRegistered autotest failing
                      • 543646: security_OpenFDs failing on veyron boards
                      • 543649: smaug canary failing paygen stage
                        CrOS gardener: stevenjb
                        • All quiet on the chrome on chrome os front.
                        Sheriff: tfiga, gedis
                        • 539720Waterfall can't accessed through Uberproxy, need to use
                        • 538744: swarming internal error on PFQ
                        • x86-generic full VMTest login_Cryptohome flake

                        2015-10-06 and 2015-10-05
                        CrOS gardener: stevenjb
                        • 539748: depot_tools update caused chromiumos.chromium builds to fail (reverted)
                        Sheriff: bsimonnet, furquan
                        • 539594: depthcharge build failure on veyron rialto [Fixed]
                        • 539748: chromeos-chrome fails to build: rmdir/mkdir access violation in (all canaries failing) [Fixed]
                        • 539720Waterfall can't accessed through Uberproxy, need to use
                        • 539739: Flaky VMTest Failures

                        CrOS gardener:
                        Sheriff: tfiga
                        • 539720Waterfall can't accessed through Uberproxy, need to use
                        • 539748: chromeos-chrome fails to build: rmdir/mkdir access violation in (all canaries failing)
                        • 539594: depthcharge build failure on veyron rialto still not fixed
                        • 538908: still flakey

                        CrOS gardener: ihf
                        Sheriff: bowgotsai
                        • 418539: Change to fixsecurity_NetworkListeners Failure is out for review.
                        • 485108: Moved desktopui_FlashSanityCheck from bvt-cq to bvt-perbuild.
                        • 538908: Lab should have recovered enough daisy_skate to continue testing.

                        CrOS gardener: ihf
                        Sheriff: bowgotsai
                        • 537655: 'Platform' object has no attribute 'SetHTTPServerDirectories'
                        • 535374: pre-cq failure: GOBError: Forbidden
                        • 538017: paygen failure (autoupdate_EndToEndTest.paygen_au_dev_delta Failure)
                        • 518591: provision Failure (DownloaderException: Could not find autotest.tar in Google Storage)
                        • 505744: provision Failure (sanity,provision,Unhandled AutoservSSHTimeout)
                        • 526453: autoupdate_Rollback Failure, ANCHOR TestFailure(au,autoupdate_Rollback,update-engine failed on chromeos4-row1-rack4-host3)
                        • 538480: autoupdate_EndToEndTest.paygen_au_canary_full
                        • 538476: autoupdate_EndToEndTest.paygen_au_canary_delta
                        CrOS gardener:
                        Sheriff: josephsih
                        • 537886: paygen failure (Failed to finish download from devserver)
                        • 530498HWTest failure (stage_artifacts timed out)
                        • 485881HWTest failure (login_OwnershipNotRetaken)
                        • 529466AUTest failure (Suite job failed or provisioning failed)
                        • 534437: amd64-generic full failed on results-40-buffet_RestartWhenRegistered
                        • 538098: Sync buildbot slave files failed: update_scripts failed
                        • 538057ERROR: Failed to stage payload: stage_artifacts timed out
                        • 538140: CQ HWTests failing in audio_CrasSanity: Unhandled AttributeError: 'Platform' object has no attribute 'SetHTTPServerDirectories'

                        CrOS gardener:
                        Sheriff: josephsih
                        • 537419: autoupdate_EndToEndTest.paygen_au_dev__full: Unhandled KeyError: 'source_payload_uri'
                        • 488291: amd64-generic ASAN failed on login_LoginSuccess
                        • 534437: amd64-generic full failed on results-40-buffet_RestartWhenRegistered
                        • 537799: PDFTestFiles/PDFExtensionTest.Load/2 and 8 are currently flaky on cros_trunk.

                        CrOS gardener:
                        Sheriff: kitching
                        • 537075: falco-full-compile paladin fails due to missing sqlalchemy import
                        • CL 301593: libgcrypt update causing problems with mesa-img
                        • 537087: lakitu_next-incremental always failing VMTest security_Minijail0 and security_SuidBinaries
                        • 409019: graphics_GpuReset failure (GPU hang) on falco-chrome-pfq
                        • 536780: added more timeouts (stage_artifacts)
                        • 536259: HW Lab Infrastructure: too few guado bvt boards (only 2, require 4)
                        • 536794: HW Lab Infrastructure: too few stumpy bvt boards (only 3, require 4)
                        • 537128: 8+ canary build trees failing on Autotest timeouts

                        CrOS gardener:
                        Sheriff: kitching
                        • 536670autoupdate_EndToEndTest.paygen_au_* test failure: Canary still has tons of argument mismatch errors even after garnold reverted his CL
                        • 536690: amd64-generic incremental build failure: updates to buffet package
                        • CL 302593: fio update fails HWTest
                        • 536515: HW Lab Infrastructure: too few auron_paine bvt boards (only 3, require 4)
                        • 536618: HW Lab Infrastructure: too few tidus bvt boards (only 3, require 4)
                        • 536775: Autotest: many builds failing due to security_ and login_ tests
                        • 536780: HWLab: builds failing due to timeouts

                        CrOS gardener:
                        Sheriff: deanliao

                        Builder x86-generic full #17275
                        • 472858 VMTest fail: DBusException: org.freedesktop.DBus.Error.NoReply
                        Several autoupdate_EndToEndTest.paygen_au_* test failures:
                        HWTest lost MySQL connection: 535795

                        CrOS gardener:
                        Sheriff: zachrmarcheu, itspeter

                        Mostly the same as 09-22. Things different from yesterday:
                        The following two CLs suspects to be root cause for Builder CQ master Build #7757 , 'NoneType' object has no attribute 'splitlines' .
                        fdeng@ had Verify-1 for those 2 CLs:
                        Chump-in CLs (Doesn't break anything, just FYI)

                        CrOS gardener: bruthig
                        Sheriff: briannorris, mylesgw, itspeter
                        • Several infra failures point to 534361; fdeng is looking at it
                        Builder Canary master Build #1307
                        • Builder jecht group canary Build #777
                          • 491290 autoupdate_Rollback failures on rikku
                        • Builder pineview group canary Build #1682
                          • autoupdate_EndToEndTest dashboard shows alex is 80%. Give it a retry.
                        • Builder auron-b group canary Build #847, Failure since 2015-09-19
                          • 533881 [paygen_au_canary] provision Failure on gandof-release/R47-7472.0.0
                          • 533879 [paygen_au_dev] provision Failure on gandof-release/R47-7472.0.0
                        Builder x86-generic full #15418
                        • 532658 VMTest fail: DBusException: org.freedesktop.DBus.Error.NoReply

                        CrOS gardener: bruthig

                        • 534437 vm_disk and vm_memory failed_SimpleTestVerify_1_autotest_tests on amd64-generic
                        • 496555 [bvt-cq] provision Failure on falco-chrome-pfq/R45-7139.0.0-rc2
                          • failing on daisy_skate
                        • 534451 PFQ builds timing out when calling /b/cbuild/internal_master/chromite/third_party/swarming.client/
                          • failed on tricky and falco
                        • 530498 [sanity] provision Failure on tricky-chrome-pfq/R47-7446.0.0-rc3
                        • 496292 [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq/R45-7137.0.0-rc2
                          • failed on lumpy
                        • 533979 [bvt-cq] audio_CrasSanity Failure on lumpy-chrome-pfq/R47-7474.0.0-rc1
                        • 478533 [bvt-cq] desktopui_FlashSanityCheck Failure on lumpy-chrome-pfq/R44-6989.0.0-rc1
                        • 418539 [bvt-inline] security_NetworkListeners Failure on daisy_skate-release/R39-6315.0.0
                        • 534544 [bvt-cq] network_DefaultProfileServices Failure on tricky-chrome-pfq/R47-7478.0.0-rc2
                        • 534399 IncidentReportingServiceTests are crashing on the cros_trunk build
                        • 510291  DevToolsPixelOutputTests.TestScreenshotRecording fails on cros_trunk official bot

                        Sheriffs: posciak, cernekee, smbarber
                        • 530203 Samus with on-going HWTest Aborts.
                        • 491290 autoupdate_Rollback failures on beltino
                        • 483749 more provision failures (daisy_skate)
                        • 409019 graphics_GpuReset failures on lumpy
                        • 533006530693 multi-GB log files from Link FSI (?) took down the lab network
                        • 488291 x86-generic, amd64-generic ASAN vmtests are failing
                        • omg/1030 unable to fetch binaries from GS

                        CrOS gardener: jonross
                        • 530646 WebViewTest.Dialog_TestConfirmDialogDefaultGCCancel Failure from V8 Roll (Also 530593)
                        • 530612 Canary master: BranchUtil: No such file or directory: '/tmp/cbuildbot-tmpd0s19p/tmprZ63D7/src/aosp/system/attestation'
                        • 518591 Veyron-group provision failure.
                        • 528748 Mario/nyan pool-health bug
                        • 530203 Samus with on-going HWTest Aborts.
                        • Actually passed last night! First pass since August 27th :D
                        • 530498 Provision failure on Tricky
                        • 471531 Peach pit failure on provision_AutoUpdate.
                        • 530605 Falco PFQ Aborting during HWTest
                        • 473976 Daisy_skate failure in HWTest
                        • 530661 Lumpy audio_CrasSanity Failure

                        CrOS gardener: jonross
                        Sheriffs: drinkcat, snanda, tbroch

                        • 530265 chromiumos sdk :: *-client pkgs need dep to chromeos-dbus-bindings.
                        • 530194 Alex and Lumpy bots failing update_scripts on missing gclient config.
                        • 529216 paygen failures now also appearing on Nyan, Rambi, and Veyron group
                        • 528748 x86-mario HWTest pool issue.
                        • 530203 Samus HWTest reported an infra issue.
                        • 471531 Daisy AutoUpdate Failure. Looks to be a flake, hopefully clears up.
                        • Other failures (Tricky, Falco) appears to be possible HWTest [bvt-inline] flakes, as the jobs timed out. No detailed logs or auto-filed bugs from failures.
                        • 530286 Falco Failure, desktopui_ExitOnSupervisedUserCrash Failure: no process matching chrome
                        • 489106 Lumpy Autotest failure during graphics_CpuReset. Unexpected termination
                        • 478416 Peach timeout during desktopui_FlashSanityCheck

                        CrOS gardener: 
                        Sheriffs: drinkcat, snanda, tbroch

                        canary failures:
                        • 529905: bobcat: setup_board fails
                        • 523313: nyan_blaze: autoupdate failed. (#2084 / #2085)
                          • Happened twice in a row now. Might be worth investigating further.....
                        • 529044: platform_Powerwash: leon does not reboot?
                          • Assigned to deymo
                        • 529216: x86-mario Paygen: Failed with Killing tasks: [<_BackgroundTask
                          • 529113: stumpy-moblab (moblab_SmokeSuite)
                            • Long standing issue
                          • 529674: canary master fails in BranchUtil
                            • Assigned to vitalybuka
                          • 529428veyron_pinky: 'str' object has no attribute 'write'
                            • Assigned to pprabhu
                          • 529565: The Paygen [stout] stage failed: <class 'chromite.lib.paygen.gslock.LockNotAcquired'>
                          • 529466: x86-mario Unhandled AutoservRunError: command execution error
                          • 529324: butterfly/sumo provision failure
                            • Race between UploadTestArtifacts and HWTest phases
                          • 529612: akitu_mobbuild: cloud_CloudInit fails in VMTest (twice)
                          • 529480: Several PFQs failing on security_SandboxLinuxUnittests
                            • Fix landed in chromium. Ensure that the subsequent runs pass.
                            • This should be fixed from 47.0.2506.0, but PFQ is not picking it up (I tried to abort the current build to force it... that did not work...). Looks like we need to wait for the release tag 47.0.2506.0 to appear on (should be Wed 8pm PST if I understand correctly).

                          CrOS gardener: 
                          Sheriffs: drinkcat, aaboagye, benzh
                          • 529480: Several PFQs failing on security_SandboxLinuxUnittests
                            • Fix landed in chromium. Ensure that the subsequent runs pass.
                          • 525128: CustomLauncherPageBrowserTest.EventsActivateSwitchToCustomPage browsertest failing on Builder: Linux ChromeOS Buildspec Tests
                            • This started failing on the 6th. Need to find a fix ASAP.
                          • 529388: ninja failed in build_packages: sys-kernel/chromeos-kernel-3_10
                            • The subsequent build seemed to work fine.
                          • 528567: daisy pool critical state (board does not recover from update to 7426.0.0, 7427.0.0 or 7428.0.0; 7429.0.0 seem ok)
                            • 529443: daisy provision_AutoUpdate failures
                          • 528748: x86-mario pool critical state => Similar to above
                            • There were some shill issues and DUTs going down. See b/23896777.
                          • 524814: daisy/x86-mario Paygen => Long standing issue. Possibly causing 2 issues above?
                            • 518591: zako/clapper: Could not find *_full_* in Google Storage (possibly same root cause)
                            • 529216: veyron_mighty/nyan_blaze: Failed with Killing tasks: [<_BackgroundTask (possibly same root cause)
                          • 529113: stumpy-moblab (moblab_SmokeSuite) => Long standing issue
                          • 487955: daisy full failure in ChromeSDK => Old issue resurfacing (missing dependency?), check if happens again on next run
                          • 529160daisy_spring failed to return from powerwash (log) => Check on next run if it happens again
                          CrOS gardener: 
                          Sheriffs: jcliang
                          • 516795: Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
                            • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
                          • 524814: Infrastructure Issues (code 3 and code 1) 
                          CrOS gardener: 
                          Sheriffs: jcliang
                          • 516795: Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
                            • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
                          • 524814: Infrastructure Issues (code 3 and code 1) 
                          • 528176: vmtest fails logging_UserCrash on x86-generic incremental, x86-generic ASAN, and amd64-generic ASAN
                          CrOS gardener: 
                          Sheriffs: cywang
                          • 516795(merged 526629): Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
                            • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
                          • 509779 Infrastructure Issues (code 3 and code 1) 
                          • 526641(fixed): after vapier clobbered ccache, sys-apps/util-linux-2.25.1-r1 built successfully
                          • 527909: veyron_jaq canaries failing to sign/paygen due to broken bsp
                          • 528017: veryon_jerry canaries failing to sign/paygen due to broken bsp
                          CrOS gardener: 
                          Sheriffs: cywang
                          • 526716 WebRTC is failing to build on mipsel. Preventing up rev of Chrome on CrOS
                          • 526629: HWTest [clapper] [bvt-inline] test timeout failure?
                            • test seems finished after the test was stopped by 'timeout'?
                          • 526641: pineview group: failed to build sys-apps/util-linux-2.25.1-r1
                          CrOS gardener: 
                          Sheriffs: dhendrix
                          • 524814: Canaries are falling over on autoupdate_EndToEndTest.paygen_au_canary_test_full
                          • Provision failures that seemed to fix themselves.
                            CrOS gardener: tbrazic
                            Sheriffs: ejcaruso, bfreed, hychao
                            • 517876: DUTs lost RPC connections
                            • 523189: login_OwnershipTaken failures on multiple boards

                            CrOS gardener:
                            Sheriffs: ejcaruso, bfreed, itspeter
                            • 522851: [Pri-0] Google cloud storage exception causing archive steps fail across major builtbot
                              • Closing tree because this cover all the errors underground.
                              • reverts which changed how we build and install rsa_id files.
                              • Note: We cannot login to bots to confirm the missing file, but the log entry "CommandException: No URLs matched: /b/cbuild/external_master/buildbot_archive/daisy-incremental/R46-7381.0.0-b24819/id_rsa" gives the best clue.
                            • 522785: buildpackages [x86-alex] [afdo_use] is flaky in pineview group canary
                            • 518591: samus-release: provision failure, infra flaky, succeed in next build.
                            • 503526: ivybridge-freon-release-group: DUTs pool is too small
                            • 496036sandybridge-release-group: DUTs pool is too small
                            • 523170: gizmo canary fails during BuildPackages
                            • 523173: nyan group canary timed out during paygen
                            • 523174: auron group canary has issues with rpm unit tests
                            • 523139: sandybridge-freon group canary, x86-alex_freon paladin not found on in master and won't build

                            CrOS gardener: girard
                            Sheriffs: dlaurie, wiley, itspeter
                            • 522533: lab problems creating various failures
                            • 522540: amd64-generic and x86-generic chromium PFQ failures in security_OpenFDs test
                            • 522528: HWTest run_suite failures
                            • 522410: ap-daemons fails to build on storm, blocking many pre-cq runs
                            CrOS gardener: girard
                            Sheriffs: dlaurie, wiley, owenlin
                            • Spontaneous network failure! Tests won't succeed without network.
                            • 522130: LKGM timeouts on Chrome PFQ (fixed)
                            • 522139: Paygen timeouts
                            • 522141: sandybridge-freon-release-group builder needs to be removed
                            • 522147: mario-incremental failing (fixed)
                            CrOS gardener: girard
                            Sheriffs: davidriley, waihong, owenlin
                            • 521642: daisy-skate build failed on database error
                            Sheriffs: davidriley, waihong, kcwu
                            • 520931: provision Failure on samus-release, looks like hardware flaky.
                            • 521046: VMTest kernel_CryptoAPI failed on lakitu canary
                            • 521018Several canary groups timed out on the step "steps" (no attribute 'PrintBuildbotStepFailure')

                            Sheriffs: jrbarnette, armansito
                            • Multiple canary failures in the AM, especially 520311.
                              • Pinned Chrome; waiting for the overnight canaries to prove whether that fixed it.

                            CrOS gardener: michaelpg
                            • 516978: No space left on device -> stateful partition sizeincreased
                            • 518015: Bots haven't signed the new CLA; LKGM candidates not uploading -> CLA enforcement rolled back
                            Sheriffs: cychiang, deymo, adlr
                            • 517308: security_test_image faild with wrong fs type to mount recovery_image.bin. --> The image seems good and running security_test_image locally can pass.
                            • 517388: mipsel-o32-generic full failed at ChromeSDK due to --hash-style defaulting to gnu
                            • 517348: [paygen_au_dev] autoupdate_EndToEndTest.paygen_au_dev_test_full Failure on peach_pi-release/R46-7335.0.0
                            • 517351: [sanity] provision Failure on lumpy-release/R46-7335.0.0 stage_artifacts timed out
                            • 517460build_package failed at chromeos-chrome
                            Chrome Gardener: stevenjb
                            • 517238: RESOLVED: ExtensionTestMessageListener::WaitUntilSatisfied() causing flake across a large number of tests
                              This turned out to actually be 515914 - browser_tests step fails even though all tests pass
                            • 516978: P0 STILL IN PROGRESS: piglit file collision cause build_image failure. --> No space left on device
                              This is causing PFQ failures
                            • 517593: Frequent browser_tests time out with (TIMED OUT) in log
                              This isn't causing any detected failures because the tests get retried, but it does slow down the tests a little and generates confusion when other issues (e.g. 
                              515914) show up.

                            Sheriffs: cychiang, dianders, denniskempin
                            • 516978: piglit file collision cause build_image failure. --> No space left on device
                            • 493752: Lab DHCP failures lead to "Host did not return from reboot". It causes PFQ build failure.
                            • 517027: jecht and veyron_pinky not available in the lab.
                            • 516795: veyron group canary is too close to its timeout. ---> seeing other boards fail of the same reason.
                            • 514700: Samus failures due to stateful partition of Samus DUTs being bad.  ext4_lookup errors, "cannot remove" errors, etc.
                            • 515880: No more samus devices in lab that are good (probably because of above bug).
                            Sheriffs: djkurtz, dianders, denniskempin
                            • 488291: Looks like vm_disk: failed_SimpleTestVerify is flaky and  throttled the tree on amd64-generic ASAN. 
                            • chromeos-hwid broke in paladin several times.  Probably <>.
                            • 516750: Samus failures due to stateful partition of Samus DUTs being bad.  ext4_lookup errors, "cannot remove" errors, etc.
                            • FYI: binutils roll happening
                            • 515528: AU tests failing.  I don't think they always ping this bug, though...
                            • 516793: cbuildbot timeouts are hard for sheriffs to decipher.
                            • 516795: veyron group canary is too close to its timeout.
                            • 530203 Samus with on-going HWTest Aborts.

                            Sheriffs: djkurtz, moch, vbendeb
                            • 516283: chromite: Unittest failure on veyron_pinky in mobmonitor/checkfile/manager_unittest: URLError: <urlopen error [Errno 111] Connection refused>
                            • 516286: chromite: Unittest failure on "auron group canary" in sync_stages_unittest timeout?
                            • 516295: zako BuildPackages fails in media-libs/fontconfig
                            • 515528[paygen_au_canary] autoupdate_EndToEndTest.paygen_au_canary_test_delta Failure on falco_li-release/R46-7315.0.0
                              • The autoupdate_EndToEndTest.* tests seem to be failing across many boards over the past ~3 days:
                            • 516371Signer test failing due to unexpected kernel parameter (CL reverted)
                            • 516430x86-alex paladin: bvt-cq timed out
                            • 516457HWTestStage fails with
                            • 488291amd64-generic ASAN failing on desktopui_ScreenLocker
                            Sheriffs: vbendeb, moch, pprabhu
                            • 5155Sheriffs: 60CrOS Tree Closure: ChromeOS killing DUTs in the lab
                            • 515905SyncChrome fails on all Chrome PFQ (non-issue - caused due Chrome pinning because of the above issue, PFQ expected to fail)
                            • 515937Temporary workarounds in lab to get DUTs off of a bad build
                                2015-07-29 and 2015-07-30
                                CrOS Gardener: alemate (29), afakhry (30-31)
                                Sheriffs: pprabhu, grundler, zuethen, 
                                  • 501178: 13 groups timeout... HWTest of many boards are not running. Closed the tree.
                                  • 515479: kvm is missing on a lot of precq bots breaking vmtest (Fixed by davidjames)
                                  • 515201: Chrome crashes (AddInputDevice) affecting other services(Fixed in Chrome; see also 515154 43375)
                                  • 514700: Samus: file system corrupted after Kingston FW update (reverted Kingston FW update)
                                  • 434755: daisydog constantly restarting (Fix in CQ - x86_64 v3.8 kernel missing /dev/watchdog since last fall)
                                  • 515576: shill crashes took a bunch of DUTs "out of service": alex/zgb/peppy/lumpy/...  (chumped Fix in shill)
                                  • HW TestLab Network failure: outbound traffic was losing 60% or more packets (subsided; still monitoring)
                                  • 515905: SyncChrome fails on all Chrome PFQ (Chrome pinned to old version?)
                                  • 515302SitePerProcessBrowserTest.DiscoverNamedFrameFromAncestorOfOpener is failing on the official cros-trunk.
                                  • 515567: Compile failures on cros-trunk due to #include the generated header "ui/gfx/vector_icons2.h".
                                  • 516052: More Flaky tests in AutofillInteractiveTest on the official cros-trunk.
                                  2015-07-27 and 2015-07-28
                                  CrOS Gardener: bruthig, alemate
                                  Sheriffs: avakulenko, abrestic, vapier/henrysu
                                  • 514257: Lost a bunch of DUTs due to AFE going down last night
                                  • 514364: coreboot branch update caused some builders to go red.
                                  • 512996 BrowserEncodingTest.TestEncodingAutoDetect timing out on cros_trunk
                                  • 513593: [bvt-inline] security_SandboxStatus Failure on lumpy-chrome-pfq/R46-7296.0.0-rc1
                                  • 514401 Multiple build's are failing syncchrome with error: "reference is not a tree: 8a0429e414914781450ca007f20b0e511b3acff7"
                                  • 514499 provision_AutoUpdate failures on Rambi
                                  • 460860 more login_Cryptohome failures
                                  • 514504 desktopui_ScreenLocker failure on Auron (flake?)
                                  CrOS Gardener: bruthig
                                  • 513593: [bvt-inline] security_SandboxStatus Failure on lumpy-chrome-pfq/R46-7296.0.0-rc1
                                  CrOS Gardener: jonross
                                  Sheriffs: quiche, shchen
                                  • 512417 New FecSendPolicy test is crashing on CrOS Trunk
                                  • 512427 Chrome PFQ failing on autotest-tests-ownershipapi
                                  • 512435 Canary board timeouts, missing boards for HWTest
                                  • 465862 Flake in desktopui_ScreenLocker, with ASAN
                                  • 509274 Canary timeout on candy
                                  • 512577 CrOS Trunk failing on ClickModifierTest
                                  • 513618 mipsel-o32-generic-full failed building Chrome

                                  CrOS Gardener: jonross
                                  Sheriffs: bleung, hungte, shawnn
                                  • 512010 Chrome OS Canary failure in BranchUtilTest failures: no such option --nobuildbottags
                                  • 512024 Master release failure
                                  • 511680 Samus HWTest failure
                                  • 491361 Strago VMTest failure
                                  • 508637 Jecht-family failure
                                  • 511542 winky DUT shortage
                                  • 512174 CrOS Commit /Queue HWTest failures: /var/log/storage_info.txt does not exist

                                  Sheriffs: bleung, shawnn
                                  • Known-issues causing multiple canary failures:
                                    • 484726: autoupdate_Rollback failing, possibly due to DNS issues in lab
                                    • 508637: rikku-family stuck at login screen
                                    • 510909: Paygen kernel hash issue. Should be resolved.
                                    • 505744: AutoservSshTimeout
                                  • 511317: login_OwnershipTaken timing out (silently?) 
                                  • 511502: libstrongswan missing symbols
                                  Sheriff: dbasehore, alecaberg
                                  • 510759: Paygen lock not acquired
                                  • 481092: builder must call SetVersionInfo first
                                  • 510909: paygen failure, kernel hash doesn't match
                                  • 509837: amd64 ASAN flake

                                  Sheriff: zqiu, marcheu, wuchengli
                                  • 510074 amd64-generic-llvm builder unittest failures
                                  • 509779 Flaky HWTest failures
                                  Sheriff: furquan, charliemooney
                                  • 465862 Flaky screen lock test
                                  • 508637 Rikku: Login screen problem in HWTest
                                  • 491290 Flaky SSH Failure

                                  Sheriff: puthik, rspangler, seanpaul
                                  • 507279 Lumpy/Falco pfq hwtest failing on timeout 
                                  • 501966 pool: bvt, board: lulu in a critical state
                                  • 507372 Drone refresh/execute took over 50s
                                  • 470701 Flaky BVT security_Firewall failure, "Mismatched firewall rules"

                                  Sheriff: posciak
                                  • 505918 CheckFileModificationTest timeouts
                                  • 506030 Payload generation failures
                                  • 506037 autotest-chrome build failures on missing dependencies

                                  Sheriff: cernekee, reinauer, kpschoedel
                                  • 505108: wolf-paladin and wolf-canary are failing, lab is closed.  Somebody will fix this Monday.
                                  • 505051: "Mismatched firewall rules" test failure on x86-generic build
                                  • 504947: HWTest failures on ivybridge, daisy
                                  • 504861: ASAN buildbot failures
                                  • 504860: HWTest did not complete due to infrastructure issues
                                  Sheriff: cernekee, reinauer, wiley
                                  • 504602: builders cannot get veyron_pinky chrome prebuilts
                                  • 504476: pre-cq builders are not running
                                  • 504400: Crash in SpokenFeedbackEventRewriterDelegate
                                  • 472895|AFDO generate| should only run if Chrome has changed
                                  Sheriff: dtor, wiley, gedis
                                  • 465862amd64-generic ASAN failure (desktopui_ScreenLocker fails - hitting regularly)
                                  • 430836: autoupdate_Rollback failure
                                  • 491598: platform_Powerwash flake
                                  • 412795: Refresh Packages is down
                                  • 488580: image_to_vm failing
                                  • 472895: Canaries failed while syncing Chrome
                                  Sheriff: bowgotsai
                                  • 488291flaky on login_LoginSuccess test
                                  • 465862amd64-generic ASAN failure
                                  • 503188: HWTest failure
                                  • 488580parrot canary: image_to_vm failing

                                  Sheriff: bowgotsai
                                  • 488291flaky on login_LoginSuccess test
                                  • 465862desktopui_ScreenLocker keeps failing on x86-generic ASAN VMTest
                                  • 502897: AU test failed on "No module named netifaces"
                                  • 502909Multiple HWTest failures
                                  • 502910CommitQueueSync failure
                                  • 503001: cbuildbot command failed on multiple PFQ builders

                                  Sheriff: josephsih
                                  • 500640Multiple HWTest failures
                                  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
                                  • 481092: ManifestVersionedSync: RuntimeError: builder must call SetVersionInfo first
                                  • 483661: x86-generic full: vmtest failed in SimpleTestUpdateAndVerify
                                  • 501178CanaryCompletion: 20 groups timed out
                                  • 444876Clear and Clone chromite: remote: User Is Over Quota

                                  Sheriff: filbranden
                                  • 500640Multiple HWTest failures
                                    • Hardware is borked and we had a deputy outage, so we had to work around it by disabling the hw_tests that were failing.
                                    • CL 277672 disabling the tests.
                                    • ChromeOS Infra team to revert that CL once the hardware is working again.
                                  • 500394build_package fails on several PFQ builtbot since 6/12
                                  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
                                    • Still open, if it keeps the tree closed we need to prioritize a revert (even possible?) or push a rushed fix.
                                  Sheriff: itspeter
                                  • 500394build_package fails on several PFQ builtbot since 6/12
                                  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
                                  Sheriff: itspeter
                                  • amd64-generic ASAN VMTest failure
                                    • 465862Build haven't been succeed since 2015-06-10, desktopui_ScreenLocker keeps failing
                                  • x86-generic ASAN VMTest failure
                                    • 488291Seems to be flaky on login_LoginSuccess test
                                    Sheriff: tyanh
                                    Gardener: girard
                                    • Infra failures on Canary bots
                                      • 491290: autoupdate_Rollback Failure on rambi-release/R45-7142.0.0
                                      • 497035: autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on link-release/R45-7142.0.0
                                      • 497059: HWTest suite prep aborted on Stumpy_moblab Canary
                                    • 497092: Chrome PFQ failing at BuildPackages across all builders - reverted patch - next build should be okay
                                      Sheriff: tyanh
                                      Gardener: girard
                                        • Infra failures on Canary bots
                                          • 496552: [au] autoupdate_Rollback; Unhandled AutoservRunError: command execution error
                                          • 496526: [sanity] provision; Unhandled HTTPError: HTTP Error 500: Internal Server Error
                                          • 476324: HWTest provision failure=
                                          • 460925: HWTest/sanity provision; Unhandled TimeoutException: Call is timed out
                                          • Host did not return from reboot
                                        • 496523 mipsel fails continuously on PFQ

                                        Gardener: jonross
                                        • 460925 Chrome PFQ seeing Infra failures in HWTest, assigned to Infra
                                          • Affecting: daisy_skate, falco, lumpy, peach_pit, 
                                        • 465162 469495 Infra failures blocking the PFQ
                                        • A series of Infra failures for Chrome OS Canary bots were auto-filled. Not enough bots in the pool
                                        • CL 274236 removed two bots from the Waterfall x86-generic-tot-chrome-pfq-informational, amd64-generic-tot-chrome-pfq-informational)
                                        • 496273 mipsel-032-generic having gyp error preventing a build.
                                        • 496293 Chrome OS PFQ is trying 45.0.2421.0 which does not contain the fix to 494912 whioh would unblock the PFQ
                                        • 496325 Flaky OAuth test on Linux Chromium OS
                                        Sheriff: robertshield, aaboagye, ssl
                                        Gardener: jonross
                                        • Chrome PFQ failures still holding up upreving.
                                          • 494041494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
                                            • Test was landed in this change assigned to developer who landed the tests
                                            • Seems like an actual regression in video playback for ARM.
                                            • Regression is from Chrome side between 45.0.2416.0 and 45.0.2417.0. Reverting the video changes in this diff does not seem to fix the issue. Trying to revert the WebRTC roll, but its failing locally. I've reached out to the WebRTC sheriff
                                            • Suspects 80f289fe303323361d07c5b58b23f8499903a154c794eda78e9ba3c46b550b433e9fe5a248d40104 as the bug is not present with hardware acceleration off. Apparently my local build has an issue, as I still have the failure with hardware acceleration off.
                                          • 413961: falco failed on stateful_update. It looks like the download got interrupted.
                                        • Canary Failures
                                          • I believe these were network related... some are reporting high flakiness
                                          • 476368495463428345 have been filed against the Canary failures.
                                          • 493219: nyan group failed with Connection reset by peer.

                                        2015-05-30/31, 2015-06-01
                                        Sheriff: ssl, aaboagye
                                        • 482284: ivybridge-freon-release-group: The BuildImage [stout_freon] [afdo_use] failed - cannot open ‘/dev/loop0p1’ for reading: Permission denied
                                        • p/40797: oak fails to cross-compile img-ddk properly
                                        • 477928: (quawks, x86-zgb) autoupdate-Rollback - ssh: Could not resolve hostname chromeos2-row2-rack7-host6: Name or service not known (network went away briefly?? DNS issue?)
                                        • 493533: asan bots failed quipper unittests. x86-generic ASAN has been broken since 5/28.
                                          • Mainly due to the quipper unittest failing, but also due to login_LoginSucess failure. See 488291.
                                          • Waiting on this CL.
                                        • Chrome PFQ failures
                                          • 494041, 494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
                                          • 494909: lumpy PFQ failing on desktopui_FlashSanityCheck.
                                        • 495281: One off wolf-tot-paladin failed during HWTest [sanity] - provision: ABORT: reboot command failed
                                        • 493718: Chrome PFQ Failing to uprev Chrome commit - on tricky, which is experimental is a mistake. Disregard these emails.
                                        • 493219: Some canaries failed due to FAILED RPC CALL. (beltino-a/b group, sandybridge-freon, ivybridge-freon) Manifested as timed out and Connection reset by peer.
                                        • CL:274726: Fix lakitu build_image error when modifying kernel command line.

                                        OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)