Sheriff Log: Chromium OS

Sheriff: posciak
  • 505918 CheckFileModificationTest timeouts
  • 506030 Payload generation failures
  • 506037 autotest-chrome build failures on missing dependencies

Sheriff: cernekee, reinauer, kpschoedel
  • 505108: wolf-paladin and wolf-canary are failing, lab is closed.  Somebody will fix this Monday.
  • 505051: "Mismatched firewall rules" test failure on x86-generic build
  • 504947: HWTest failures on ivybridge, daisy
  • 504861: ASAN buildbot failures
  • 504860: HWTest did not complete due to infrastructure issues
Sheriff: cernekee, reinauer, wiley
  • 504602: builders cannot get veyron_pinky chrome prebuilts
  • 504476: pre-cq builders are not running
  • 504400: Crash in SpokenFeedbackEventRewriterDelegate
  • 472895|AFDO generate| should only run if Chrome has changed
Sheriff: dtor, wiley, gedis
  • 465862amd64-generic ASAN failure (desktopui_ScreenLocker fails - hitting regularly)
  • 430836: autoupdate_Rollback failure
  • 491598: platform_Powerwash flake
  • 412795: Refresh Packages is down
  • 488580: image_to_vm failing
  • 472895: Canaries failed while syncing Chrome
Sheriff: bowgotsai
  • 488291flaky on login_LoginSuccess test
  • 465862amd64-generic ASAN failure
  • 503188: HWTest failure
  • 488580parrot canary: image_to_vm failing

Sheriff: bowgotsai
  • 488291flaky on login_LoginSuccess test
  • 465862desktopui_ScreenLocker keeps failing on x86-generic ASAN VMTest
  • 502897: AU test failed on "No module named netifaces"
  • 502909Multiple HWTest failures
  • 502910CommitQueueSync failure
  • 503001: cbuildbot command failed on multiple PFQ builders

Sheriff: josephsih
  • 500640Multiple HWTest failures
  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
  • 481092: ManifestVersionedSync: RuntimeError: builder must call SetVersionInfo first
  • 483661: x86-generic full: vmtest failed in SimpleTestUpdateAndVerify
  • 501178CanaryCompletion: 20 groups timed out
  • 444876Clear and Clone chromite: remote: User Is Over Quota

Sheriff: filbranden
  • 500640Multiple HWTest failures
    • Hardware is borked and we had a deputy outage, so we had to work around it by disabling the hw_tests that were failing.
    • CL 277672 disabling the tests.
    • ChromeOS Infra team to revert that CL once the hardware is working again.
  • 500394build_package fails on several PFQ builtbot since 6/12
  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
    • Still open, if it keeps the tree closed we need to prioritize a revert (even possible?) or push a rushed fix.
Sheriff: itspeter
  • 500394build_package fails on several PFQ builtbot since 6/12
  • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
Sheriff: itspeter
  • amd64-generic ASAN VMTest failure
    • 465862Build haven't been succeed since 2015-06-10, desktopui_ScreenLocker keeps failing
  • x86-generic ASAN VMTest failure
    • 488291Seems to be flaky on login_LoginSuccess test
    Sheriff: tyanh
    Gardener: girard
    • Infra failures on Canary bots
      • 491290: autoupdate_Rollback Failure on rambi-release/R45-7142.0.0
      • 497035: autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on link-release/R45-7142.0.0
      • 497059: HWTest suite prep aborted on Stumpy_moblab Canary
    • 497092: Chrome PFQ failing at BuildPackages across all builders - reverted patch - next build should be okay
      Sheriff: tyanh
      Gardener: girard
        • Infra failures on Canary bots
          • 496552: [au] autoupdate_Rollback; Unhandled AutoservRunError: command execution error
          • 496526: [sanity] provision; Unhandled HTTPError: HTTP Error 500: Internal Server Error
          • 476324: HWTest provision failure=
          • 460925: HWTest/sanity provision; Unhandled TimeoutException: Call is timed out
          • Host did not return from reboot
        • 496523 mipsel fails continuously on PFQ

        Gardener: jonross
        • 460925 Chrome PFQ seeing Infra failures in HWTest, assigned to Infra
          • Affecting: daisy_skate, falco, lumpy, peach_pit, 
        • 465162 469495 Infra failures blocking the PFQ
        • A series of Infra failures for Chrome OS Canary bots were auto-filled. Not enough bots in the pool
        • CL 274236 removed two bots from the Waterfall x86-generic-tot-chrome-pfq-informational, amd64-generic-tot-chrome-pfq-informational)
        • 496273 mipsel-032-generic having gyp error preventing a build.
        • 496293 Chrome OS PFQ is trying 45.0.2421.0 which does not contain the fix to 494912 whioh would unblock the PFQ
        • 496325 Flaky OAuth test on Linux Chromium OS
        Sheriff: robertshield, aaboagye, ssl
        Gardener: jonross
        • Chrome PFQ failures still holding up upreving.
          • 494041494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
            • Test was landed in this change assigned to developer who landed the tests
            • Seems like an actual regression in video playback for ARM.
            • Regression is from Chrome side between 45.0.2416.0 and 45.0.2417.0. Reverting the video changes in this diff does not seem to fix the issue. Trying to revert the WebRTC roll, but its failing locally. I've reached out to the WebRTC sheriff
            • Suspects 80f289fe303323361d07c5b58b23f8499903a154c794eda78e9ba3c46b550b433e9fe5a248d40104 as the bug is not present with hardware acceleration off. Apparently my local build has an issue, as I still have the failure with hardware acceleration off.
          • 413961: falco failed on stateful_update. It looks like the download got interrupted.
        • Canary Failures
          • I believe these were network related... some are reporting high flakiness
          • 476368495463428345 have been filed against the Canary failures.
          • 493219: nyan group failed with Connection reset by peer.

        2015-05-30/31, 2015-06-01
        Sheriff: ssl, aaboagye
        • 482284: ivybridge-freon-release-group: The BuildImage [stout_freon] [afdo_use] failed - cannot open ‘/dev/loop0p1’ for reading: Permission denied
        • p/40797: oak fails to cross-compile img-ddk properly
        • 477928: (quawks, x86-zgb) autoupdate-Rollback - ssh: Could not resolve hostname chromeos2-row2-rack7-host6: Name or service not known (network went away briefly?? DNS issue?)
        • 493533: asan bots failed quipper unittests. x86-generic ASAN has been broken since 5/28.
          • Mainly due to the quipper unittest failing, but also due to login_LoginSucess failure. See 488291.
          • Waiting on this CL.
        • Chrome PFQ failures
          • 494041, 494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
          • 494909: lumpy PFQ failing on desktopui_FlashSanityCheck.
        • 495281: One off wolf-tot-paladin failed during HWTest [sanity] - provision: ABORT: reboot command failed
        • 493718: Chrome PFQ Failing to uprev Chrome commit - on tricky, which is experimental is a mistake. Disregard these emails.
        • 493219: Some canaries failed due to FAILED RPC CALL. (beltino-a/b group, sandybridge-freon, ivybridge-freon) Manifested as timed out and Connection reset by peer.
        • CL:274726: Fix lakitu build_image error when modifying kernel command line.
        Sheriff: dianders
        • 493718: Chrome PFQ Failing to uprev Chrome commit - on tricky, which is experimental?
        • 493301: master-paladin has encountered infra failures - turned out to be hang in bh_submit_read again.
        • 493730master-paladin has encountered infra failures - shouldn't have been blamed on infra when there's a kcrash.
        • 493752 (AKA 493752): slippy-release-group: The HWTest [leon] [sanity] stage failed - HWTest did not complete due to infrastructure issues (code 3) - DHCP problems
        • 493144 x3: sandybridge-freon-release-group: The HWTest [stumpy_freon] - HWTest did not complete due to infrastructure issues  - pool: bvt, board: stumpy_freon in a critical state
        • 428345 (AKA 493752): beltino-a-release-group: The AUTest [mccloud] [au] stage failed - HWTest failed (code 1) - platform_Powerwash - ABORT: Host did not return from reboot - DHCP problems
        • 430836 (AKA 493752): beltino-a-release-group: The AUTest [mccloud] [au] stage failed - HWTest failed (code 1) - autoupdate_Rollback - ABORT: Host did not return from reboot - DHCP problems
        • 493796: RlzLibTests failures on the official cros_trunk.
        • 493811nyan-release-group: The HWTest [nyan_blaze] [bvt-inline] stage failed - HWTest failed (code 1) - Best guess is that a failed prev test put host in bad state (??)
        • 460925 (aka 493219) x4: Several (winky, quawks_freon, tidus, lumpy) - HWTest did not complete due to infrastructure issues (code 3) - Autotest FAILED RPC CAL
        • 491968 (aka 493219): auron_yuna - - HWTest did not complete due to infrastructure issues (code 3) - Autotest FAILED RPC CAL
        • 482284: ivybridge-freon-release-group: The BuildImage [stout_freon] [afdo_use] failed - cannot open ‘/dev/loop0p1’ for reading: Permission denied
        • p/40797: oak fails to cross-compile img-ddk properly
        Sheriff: dianders
        • 493176: pre-cq failing due to chromeos-base/ap-daemons
        • 493207: (warning) chromeos-base/ap-daemons failing on 1st try (old bug)
        • Lots of failures at all the same time with RPC failures talking to autotest
          • 493192The Paygen [auron] stage failed - Error sending request to Autotest RPC proxy: ... EOF occurred in violation of protocol
          • 493192: The Paygen [butterfly] stage failed - Error sending request to Autotest RPC proxy: ... Connection reset by peer
          • 493192: slippy-release-group: The Paygen [peppy] stage failed and The Paygen [leon] stage failed - Error sending request to Autotest RPC proxy: ... Connection reset by peer
          • 493192: rambi-a-release-group: The HWTest [clapper] [sanity] stage failed - Error sending request to Autotest RPC proxy ... Connection reset by peer
          • 493192: sandybridge-freon-release-group: The Paygen [lumpy_freon] stage failed - Error sending request to Autotest RPC proxy: ... Connection reset by peer
        • 463145jecht-release-group: The Paygen [rikku] stage failed - Host did not return from reboot
        • 493251: master-paladin infra failures: more autotest flakiness; letting build deputy handle this
        • 493251rambi-c-release-group: The HWTest [candy] [bvt-inline] stage failed - Suite timed out before completion - Job says ABORT: Timed out, did not run.
        • 493273: The Paygen [link] stage failed - paygen_au_canary_test_full    [ FAILED ] - DUT just seemed to die.  Baffling.
        • 493301: The Paygen [stout_freon] stage failed - paygen_au_canary_test_delta - task update_engine:1209 is hung for 60 seconds during shutdown.
        • 482284: rambi-b-release-group: The BuildImage [glimmer] [afdo_use] stage failed - cannot open ‘/dev/loop2p1’ for reading: Permission denied
        • 493326: The UnitTest stage failed: upload_symbols_unittest
        • 493330: The UnitTest stage failed: simple_builders_unittest
        • Lots of failures with more RPC failures:
        • 477928 (aka 490460): quawks-release (and daisy-release-group): The AUTest [au] stage failed - Could not resolve hostname chromeos4-row10-rack8-host17: Name or service not known 
        • 471518 (aka 490460): daisy-release-group: The AUTest [daisy_skate] - Could not resolve hostname chromeos2-row5-rack4-host2: Name or service not known
        • 493129 - jecht-release-group: The HWTest [rikku] [bvt-inline] stage failed - Too many dead rikku boards?
        • 466919 (aka 467066) - beltino-b-release-group: The Paygen [monroe] stage failed - symlink has no referent (in rsync)
        • 493533: asan bots failed quipper unittests
        • CL:273724: daisy public builders all failing due to mali-drivers-bin manifest mismatch

        Sheriff: waihong, dhendrix, rongchang
        • 428345beltino-a group canary and slippy group canaryplatform_PowerWash failed to reboot which is flaky recently
        • 492161: daisy-group canary: autoupdate_Rollback: DUT pingable but not sshable
        • 491479sandybridge-freon group canary: Not enough parrot_freon avaiable
        • veyron group canary: only jerry build_packages failed in symbol refined on llseek.c of util-linux package
        • auron group canary: cbuildbot updating slave build timed out on several recent builds
        • 465862: desktopui_ScreenLocker failure happened again on amd64-generic ASAN
        • 483661: vmtest failure in SimpleTestUpdateAndVerify happened again on x86-generic full
        • 492281"MySQL server has gone away" happened in CQ  and canaries.

        Sheriff:  chinyue

        Sheriff:  owenlin, gwendal
          • 434201security_SandboxLinuxUnittests_SERVER_JOB Failure
          • CL:*216896: whirlwind canary failing signer tests due to new kernel option
          • 490057: hwlab burned down
          • Chrome PFQ failing on x86 bots with use_sysroot error -> already fixed in Chrome; waiting for new LKGM on Chrome side to include fix
          • 490546: ImageTest fails on mips bots due to unresolved symbols in mesa; CL:272365 reverted in the mean time
          • 491012: chrome pfq build failure in
          • 491103: samus paladin crashed while talking to cidb
          • CL:217315: fixed uprev/egencache/chromeos-oak errors
          Sheriff:  tbroch, gwendal, jchuang
            • 483661x86-generic full: vmtest failed in SimpleTestUpdateAndVerify
            • 489733: veyron canary failed to build openssh.  Missing symbols from netdb ... part of glibc 
            • 472858: x86-generic ASAN, login_OwnershipApi:  fails when dbus "Did not receive a reply"
            • 477928: jecht-release-group: tidus: autoupdate_Rollback: unable to find host ... network flake?
            • 428345auron-b-release-group: lulu: platform_Powerwash,Host did not return from reboot
            Sheriff:  tbroch, gwendal, jchuang
            • 489302: peach-pits failing in lab.  Loose physical connection for entire rack in lab led to devices discharging.
            • 488644slippy-release-group: The AUTest [falco_li] [au] stage failed, 
            • 489215PFQ Master: binhosttest fail: AssertionError - cannot find Chrome prebuilts

              Sheriff:  itspeter, pstew, ejcaruso
              • 488539: oak canary was moved out of experimental and failed due to a large non-bisectable audio patch series going into 3.18
              • 488481: devserver fell over during a test, but it seems like a transient failure
              • 488580: image_to_vm failing; jrbarnette@ trying to find an owner
              • 488487488635: chrome crashes, achuith@ trying to find out how to roll back, pstew@ pinned chrome to 44.0.2401.3_rc-r1
              • 488644: ivybridge-freon failed during paygen step
              • 488739: bad X11 header include is breaking Chrome PFQ for freon images
              • 488959: buffet unittests failing on amd64 asan

              Sheriff: djkurtz, pstew, ejcaruso
                • 487955: veyron_pinky ChromeSDK fail - - ui/resources/grit/ui_resources.h: No such file or directory
                • 487959: nyan_blaze BuildPackages fail - chromeos-base/chromeos-factory
                • 487990: All chromiumos.chromium builders broken by video_encode_accelerator_unittest
                • 487997chrome PFQ - BuildPackage fails - chromeos-chrome-44.0.2401.0_rc-r1 - - ui/events/keycodes/dom3/dom_code.h: No such file or directory
                  • Suspect Chrome CL
                • 461087: Chrome PFQ fails with login_OwnershipNotRetaken HWTest failure on Tricky -- sheriffs get email on this with strong verbiage that Chrome OS won't be picking up a new Chrome for this reason, but it turns out that the PFQ master ignores this build failure because Tricky is experimental.
                • b/21165021: Canaries fail if they tried to connect to databases/servers between 2:45pm and 3:00pm PST due to power brownout
                • 488291: x86-generic ASAN failure in login_LoginSuccess
                • 488308: various paladin builders failing due to infrastructure issues


                Sheriff: wiley, puneetster

                • 460860: login_Cryptohome Failure on tricky_freon-release
                • 487120squawks boards in short supply in BVT pools, hwtests won't finish in time
                • 487303: 503 killed suite job during Leon canary
                • 433204: autoupdate_EndToEndTest.paygen_au_canary_test_full fails "... no attribute 'Error'"
                • 487772: amd64-generic ASAN builder has been red since April 6?!?

                Sheriff: cychiang

                • 486497: paygen failing: "Payload integrity check failed: install_operations[615](REPLACE_BZ): src_length is zero."  Revert CL
                • 465711: Host did not return from reboot. This also breaks PFQ
                • 487055: stout canary: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **
                • 487123: Canary builders got killed by signal 9 around May 11 22:58:31 2015 buildbot time.

                Sheriffs: garnold, jwerner
                Gardener: stevenjb
                • 476649: Sandybridge canary failure due to lumpy DUT connectivity issue.
                • b/20859395: Stumpy moblab canary failed on hwtest.
                • b/20859821: Daisy canary failed on hwtest.
                • Reverted: (suspected causing flaky interactive_ui_tests failure)
                • 482446: Sporadic failures
                • 477889: Sporadic failures
                • 484307: Still in-progress. Much time spent trying to track this down.
                • 484726: Multiple canary failures due to lab DNS flake.
                • CL:268999: Stout vmtest failing due to bad CL.
                • 484243: Multiple canary failures due to lab test fail on ACL error.
                • 486497: most canaries dying in paygen with "install_operations[1219](REPLACE_BZ): src_length is zero."

                Sheriffs: posciak
                • 476550 ivb failures on TestBlacklistedFileTypes, TestValidInterpreter, etc.
                • 482905 slippy ABORT: reboot command failed
                • 477739 samus suite prep abort

                Sheriffs: snanda, benzh, hungte
                • 483174: setuid bit missing on ping, breaking ping in chroot
                • 476649Some lumpy duts unreachable
                • 482956PFQ Master: AssertionError - cannot find Chrome prebuilts
                • 482905: provision: ABORT: Host did not return from reboot
                • 433389: [Samus] Kernel crash meta files shown after rebooting device
                • 463861: [bvt-inline] provision Failure on lumpy-release/R42-6812.15.0
                • 482454: Delta to itself failed with NewRootfsVerificationError in veyron_speedy-release/R44-7019.0.
                • CL:*215145 storm premp rejected by signer due to security checks forced off
                Gardener: afakhry
                • 482121 TabindexSaveFileDialog/FileManagerBrowserTest.Test/2 has been failing on the official cros trunk.
                • 409019 graphics_GpuReset Failure on falco PFQ.
                • 482171 graphics_Sanity_SERVER_JOB Failure on falco PFQ.
                • 482164 login_LoginSuccess Failure on lumpy PFQ.
                • 481732 Flakey provision failure on falco PFQ.
                • 482454 veyron_speedy failing AU test
                • CL:*215145 storm premp rejected by signer due to security checks forced off
                Gardener: afakhry
                • 481544 Chrome PFQ is failing on x86-generic, alex, lumpy, peppy and tricky due to desktopui_ScreenLocker VMTest failures.
                • 481820 bluez unittests breaking on amd64-generic
                • 481864 asan bots failing unittests: error adding symbols: DSO missing from command line
                • 481872 amd64 asan bot failing unittest due to leaks in libchromeos
                Gardener: derat
                Sheriffs: posciak
                • 466719 lumpy PFQ failures "Unhandled AutoservSSHTimeout"
                • 465178 tricky PFQ failed 3 times due to a login timeout in telemetry_LoginTest, but then passed again
                • 476584 (possibly) samus-canary failure, issues in TestBlacklistedFileTypes
                • More ASAN failures, looks like 478605 not fixed after all?
                • 480638 MenuControllerMnemonicTestMnemonicMatch.MnemonicMatch failing on cros trunk
                • 480667 SSLUITest.BadCertFollowedByGoodCert failing on cros trunk
                • 470130 Chromium OS (x86) Asan builder still always failing with MySQL error (one month and counting!) 
                Gardener: derat
                Sheriffs: moch, wuchengli
                  • 463805 PFQ lumpy run failed with "AutoservSSHTimeout: ('ssh timed out', * Command: )"
                  • 480491 TabindexOpenDialog_FileManagerBrowserTest browser_tests failing on cros trunk
                  • 470130 Chromium OS (x86) Asan builder failing with "Can't connect to MySQL server on '' (110)"
                  • 478605 Pretty much all tests failing on Chromium OS (amd64) Asan builder (at least, I hope this is the cause)
                  • 480514 lxc container failed to get IP address from bridge
                  • 480638 MenuControllerMnemonicTestMnemonicMatch.MnemonicMatch failing on cros trunk
                  • 480667 SSLUITest.BadCertFollowedByGoodCert failing on cros trunk
                  • 465862 desktopui_ScreenLocker failing on Chromium OS (amd64) Asan
                  • 449361 HWTest did not complete due to infrastructure issues (code 3)" on enguarde
                  Sheriffs: moch, tyanh
                  2015-04-20, 2015-04-21
                  Sheriffs: dbasehore, alecaberg, tyanh
                  • 476550 paygen issue for signing images
                  • 477739 bvt suite abort
                  • 463805 Provision failure on SSH timeouts
                  • 478713 Autotest execution errors
                  • 478605 CrOs asan failures
                  • 478762 mod-image-for-recovery failure on non-x86
                  Sheriffs: semenzato, quiche, mcchou (shadow)
                  • 477747 ImageTest failure on canries due to /usr/include/tiffio.hxx, /usr/libexec/perf-core/tests/, and /lib64/
                  • 477883 kayle-paladin failed due to chromeos-initramfs
                  • 477888 CQ failure due to VM test timeout in buffet_InvalidCredentials
                  • 477889 CQ failure due to VM test timeout in buffet_Registration
                  • 477941 link pre-cq failure (with no CLs): build_image failed due to no space on loop device
                  • 473970 canary failure due to autoupdate_EndToEndTest
                  • 474831 moblab_runsuite failure: /root/.boto did does not exist
                  • 478605 asan bots both dead in vmtest w/new chrome
                  • CL:266444 non-x86 archive failures in mod_image_for_recovery
                  Sheriffs: semenzato, quiche, mcchou (shadow)
                  • 477703 broke Archive step
                  • 477712 enabling frame pointers broke optimized webrtc code in chrome
                  • 477739 samus-release failed in suite prep
                  Sheriffs: avakulenko, jrbarnette, fjhenigman
                  • 476434 Network flakiness causes a dozen canaries go red.
                    • As of this evening, the problem is still causing intermittent failures across the board.
                  • 477352 Chrome fails to start on jerry and mighty
                    • Early morning: Chrome has been pinned to 44.0.2368.0 until the problem is fixed.
                    • Mid-day: Testing showed the problem is on the Chrome OS side; reverted this CL.
                    • Afternoon: Chrome is unpinned.
                    • Evening: Waiting for the CL to make it through a canary, so that the veyron builders will show green.
                  Gardener: jonross
                  Sheriffs: avakulenko, jrbarnette, kpschoedel
                  • 438729 OutOfProcessPPAPITest.MediaStreamAudioTrack flaky on CrOS. Taking out the X server, causing other tests to fail.
                  • 469119 TouchExplorationTest.RewritesEventsWhenOn is flaky
                  • 476934 Chrome PFQ bots failing with config failures, blocking uprev.
                  • 475923 TouchExplorationTest.SplitTapExplore flaking 50% of the time.
                  • 476550 Paygen signing failure on multiple boards (including veyron)

                  2015-04-10 and  2015-04-13
                  Gardener: jonross
                  Sheriffs: kpschoedel, denniskempin, bleung
                  • 476607 Peach_pit Chrome PFQ failng HWTest step due to Infra issue.
                  • 476577 KioskUpdateTests are flaky on Linux Chromium OS.
                  • 469119 Flaky TouchExplorationTest.RewritesEventsWhenOn is causing failures on CrOS trunk
                  • 476338 PFQ Failure on lumpy. Provision failure tracking SSH timeout.
                  • 476434 Tree is throttled, devservers are causing a P0 issues causing AU testing to flake heavily for the morning PST
                  Gardener: zork
                  • 475170: Autotest security_OpenFDs failing.

                  2015-04-06 and 2015-04-07
                  Gardener: xiyuan
                  Sheriffs: deymo, shawnn, vapier, deanliao
                  • 474227: Lack of speedy DUTs in lab
                  • 465862: desktopui_ScreenLocker failing on asan bots
                  • br:764: buffet autotests flaking on amd64-generic-full
                  • 474497: Canaries dead in paygen
                  2015-04-02 and 2015-04-03
                  Sheriffs: wfrichar, adlr, itspeter (on holiday), vapier
                  • cbuildbot gsutil errors, solved by fixing permissions manually (by system services team) and
                  • Unittest break in germ, quickl fixed by Jorge
                  • repo upload failing on kernel repo: b/20062832b/19932429
                  • CL:263970 gnutls pulled due to new pyshark code (CL:262457)
                  • 473738: sumo board failing to build adhd
                  • 473742: rush_ryu recovery kernel failing
                  • br:344: kernel git checkouts failing on gizmo/project-sdk bots
                  • CL:*211751: nyan_freon failing signer tests
                  • 473721: paygen ran out of space on swanky
                  • 473899: paygen not finding images
                  • 473900: mips qemu fc-cache call crashing

                  2015-03-31 and 2015-04-01
                  Sheriffs: bfreed, bhthompson, josephsih
                  • Throttled tree in the morning, most canaries red.  471656: Paygen: failed to set up loop device
                  • 472378: x86 asan builder failing unittests due to leak san not being supported
                  • 472658: Paygen fails: Permission denied: '/dev/loop27p4'
                    • The reverts above get the canaries past "failed to set up lop device", but many now fail with "Permission denied: '/dev/loop27p4'".
                  • 449738: "Internal server error" should be treated as infrastructure failure so it doesn't reject CLs
                    • "CQ encountered a critical failure" email.
                    • Pointer to shows most failures due to "FAILED RPC CALL: get_jobs" in HWTest.
                  • 347423: Gerrit failed to submit change.
                    • Sounds like a transient failure.  Should just try it again.
                    • 13:57:51: WARNING: Change 263241 was submitted to gerrit without errors, but gerrit is reporting it with status "NEW" (expected "MERGED").
                    • 13:57:51: ERROR: Most likely gerrit was unable to merge change 263241.
                  • 472858: VmTest fails when dbus "Did not receive a reply"

                  2015-03-30 and 2015-03-31
                  Sheriffs: josephsih
                  • 471656: Paygen: failed to set up loop device: No such file or directoryIs. This is a new issue causing the failure of 20 canary builders.
                  • 466777: HW_Test error: this issues still exists.
                  • 469259: fails: this issue still exists.

                  2015-03-26 and 2015-03-27
                  Gardener: skuhne
                  Sheriffs: puthik, furquan
                  • PFQ failed to uprev. Caused by 991533002, reverted (Chrome and ChromeOS PFQ).
                  • 469566: pool: bvt, board: veyron_jerry in a critical state
                  • 466777: pool: bvt, board: veyron_speedy in a critical state
                  • 469259: fails with "partx: /dev/loop0: error deleting partition ..."
                  • 471032: PFQ still fails - now: The binary size of the image exceeds the limits. (e.g. Daisy-freon, Daisy-skate)
                  • 449766PFQ still fails - now: [bvt-inline] login_LogoutProcessCleanup Failure on Falco

                  2015-03-24 and 2015-03-25
                  Gardener: jamescook
                  Sheriffs: charliemooney, kcwu
                  • 463530461087  Chrome uprev failure on tricky, cryptohome not mounted
                  • 466719 lumpy ssh timeout resulting in provisioning failures
                  • 470172 MasterUploadPrebuilts failure in Update PFQ config dump
                  • 470118 Amd64-generic failing vmtest "buffet_Registration" with urlopen "Connection Refused"
                  • 470237: WallpaperManagerBrowserTest.DevLaunchApp failing on cros_trunk official builders
                  • 470130: Chromium OS (x86) Asan always failing with MySQL connection error
                  • 470381: [bvt-cq] graphics_SanAngeles Failure on tricky-chrome-pfq/R43-6910.0.0-rc5 - wflinfo / waffle problem, test disabled
                  • 1035763003: Revert of Test Accelerators In Interactive UI Tests - cros_trunk official builder was failing
                  • 448247: [bvt-inline] provision Failure on candy-release/R42-6683.0.0 - update_engine failed
                  • 470701: Flaky BVT security_Firewall failure, "Mismatched firewall rules"
                  2015-03-18 and 2015-03-19
                  Gardener: abodenha
                  Sheriffs: stevefung
                  • 468340, 468770: PFQ and canaries are a mess due to infra issues
                  • 465862: amd64 ASAN builds are failing
                  • 468394: autoupdate_EndToEndTest.paygen_au_stable_test_delta failures
                  • 466777: not enough duts
                  • 467975: Image Signing Timeouts
                  • 463805:  autoserv timeouts
                  Sheriffs: littlecvr,stevefung

                  • 467975: Image Signing Timeouts
                  • 260602: Fix auron_paine build break

                  Sheriffs: littlecvr,dlaurie,zqiu

                  • 466972: insufficient DUTs for butterfly
                  • 466777: insufficient DUTs for veyron_speedy
                  • 465230: wolf: login_OwnershipTaken_SERVER_JOB Failure
                  • 433970: wolf: login_LogoutProcessCleanup_SERVER_JOB Failure
                  • 419772: gnawty: security_ProfilePermissions_SERVER_JOB Failure
                  • 411608: gnawty: security_NetworkListeners_SERVER_JOB Failure
                  • 434148: gnawty: login_MultiUserPolicy_SERVER_JOB Failure

                  Sheriffs: tyanh,dlaurie,zqiu

                  • 426164: [au] autoupdate_EndToEndTest.npo_test_delta Failure on nyan_blaze-release/R38-6158.71.0
                  • 466919: [paygen_au_canary] autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on quawks-release/R43-6872.0.0
                  • 8 issues on [bvt-inline] login_* Failure on wolf-release/R43-6872.0.0
                    • 434202: login_RetrieveActiveSessions_SERVER_JOB Failure
                    • 434182: login_SameSessionTwice_SERVER_JOB Failure
                    • 434178: login_OwnershipNotRetaken_SERVER_JOB Failure
                    • 434185: login_Cryptohome_SERVER_JOB Failure
                    • 419772: security_ProfilePermissions_SERVER_JOB Failure
                    • 434148: login_MultiUserPolicy_SERVER_JOB Failure
                    • 403701: login_MultipleSessions_SERVER_JOB Failure
                    • 434195: login_GuestAndActualSession_SERVER_JOB Failure
                  • b/19729024: incorrect DHCP config change was pushed breaking DHCP in the lab
                  • br/590: amd64 asan failing unittests due to leaks related to protobuf

                  Sheriffs: tyanh

                  • 462734: [sanity] provision Failure on lumpy-release/R43-6869.0.0

                  Sheriffs: chirantan, thieule
                  Gardener: oshima

                  • 465752: [bvt-inline] login_RetrieveActiveSessions Failure on tricky
                  • 465963: daisy canary: The AUTest [daisy_spring] [au] stage failed: ** HWTest did not complete due to infrastructure issues (code 3)
                  • 464171: Multiple canary's are failing due to kernel size limits
                  • 465877: [bvt-inline] login_OwnershipTaken Failure on x86-mario-release/R43-6865.0.0
                  • 464751: [au] provision Failure
                  Sheriffs: chirantan, thieule
                  Gardener: achuith

                  • 464938: Test lab performance issue
                  • 465596: update_engine is failing on all canary builders
                  Sheriffs: benchan, tbroch
                  Gardener: tengs

                  Sheriffs: benchan, tbroch
                  Gardener: tengs
                  • 464407 Your "Oauth 2.0 User Account" credentials are invalid .... Failure: Invalid response 302..

                  Sheriffs: dhendrix, waihong
                  Gardener: derat

                  Sheriffs: dhendrix, waihong
                  Gardener: derat
                  • 462842 Chromium OS (amd64) Asan bot failing unittests due to webserver leaks
                  • 463493 Flaky failure in breakpad's linux_client_unittest on X86 (chromium)
                  • 463532 browser_tests failing on cros trunk in WebViewTest.FileSystemAPIRequestFromWorkerDeny
                  • 464053 Beaglebone kernel too big, causing build failures
                  • 463411 pool: bvt, board: leon in a critical state. (Some DUTs had their USB ethernet dongles swapped and were down for a bit)
                  Sheriffs: dgreid, tbroch
                  Gardener: derat
                  • 461406 Chromium OS (x86) Asan bot still failing
                  • 458775 peach_pit nightly chrome PFQ failed with too few available DUTs
                  • 463213 X86 (chromium), Daisy (chromium), and AMD64 (chromium) failing on chromeos-chrome with "ValueError: invalid literal for int() with base 10"
                  Sheriffs: dgreid, tbroch
                  Sheriff: dianders, pstew, jchuang
                  • 462240 [storm-release canary] storm-release: The BuildPackages stage failed: Packages failed in ./build_packages
                  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
                  • 461841 rambi-c-release-group: timed out

                  Sheriff: dianders, pstew, tyanh
                  • 461184 [canary] HWTest did not complete due to infrastructure issues again in canary 670; also "peach-release-group: timed out"
                  • 461841 [canary] sandybridge-release-group: The Paygen [butterfly] stage failed: <class 'chromite.lib.timeout_util.TimeoutError'>: Timeout occurred- waited 13800 seconds
                  • 461893 [canary] rambi-a-release-group: The HWTest [expresso] [bvt-inline] stage failed: ** Suite timed out before completion **
                  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
                  Sheriff: amstan, gwendal, tyanh
                  • 461184 [canary, chrome pfq] HWTest did not complete due to infrastructure issues (HWTest Lab closed?)
                  • 461188 [pineview canary] Operation timed out at the end of a build; transient, subsequent build succeeded.
                  • 438908 Image signing timed out across many platforms on canary; transient, subsequent builds succeeded.
                  • 461378 chrome compilation error prevents some build to complete in CQ.
                  • 461415 critical fix in chrome was only present on chrome ToT.

                  Sheriff: amstan, gwendal  Gardener: flackr
                  • 460815 Cros SDK broken in .bashrc_profile
                  • 460693 Chrome PFQ failing with exception: global name 'AccessDeniedException' is not defined in File "src/build/", line 71
                  • 460951 ChromeMetricsServiceAccessorTest.MetricsReportingEnabled, ExternalCacheTest.Basic, ExternalCacheTest.PreserveInstalled, DeviceLocalAccountExternalPolicyLoaderTest.ForceInstallListSet failing on cros_trunk
                  • 458122 Reopened FileSystemProviderApiTest.BigFile failing on cros_trunk as test is failing 98% of runs on cros_trunk
                  • 461021 MetricsServicesManagerTest.GetRecordingLevelCoarse faililng on cros_trunk.
                  • 461046 MetricsServicesManagerTest.GetRecordingLevelFine faililng on cros_trunk.
                    Sheriff: posciak, namnguyen, snada
                    • Note to PST sheriffs: lots of infrastructure failures (I think) over the last few days, which I don't really understand. I think we need help from Infra team before we can reopen the tree for good.
                    • 419904 IndexError: list index out of range on moblab_RunSuite 
                    • 458613 Pre-CQ Launcher failures
                    • 224 Moblob failures
                    • 459679: Moblab blocks canaries
                    • b/19426205: Missing commit info

                    Gardener: jonross
                    • Reverted a change that was failing a compile on Chrome OS.
                    • 458567 Disable flaky InProcessAccessibilityBrowserTest.VerifyAccessibilityPass
                    • 458549 Disable flaky TextInput_TextInputStateChangedTest.SwitchingAllTextInputTest
                    • 458526 Chromium OS Waterfall bots falling on LKGMSync
                    • 458918 amd64 asan failing shill unittests due to leaks
                    Gardener: jonross
                    • 458341 Disable flaky LoginPromptBrowserTest.LoginInterstitialShouldReplaceExistingInterstitial
                    • 458333 Disable flaky AutofillDialogControllerSecurityTest.DoesntWorkOnBrokenHttps
                    • Reverted patch that broke Chrome OS LoginUITests
                    • 458154 Chrome PFQ peach_pit. HWTest failure, RPC Connection Timeout.
                    • 458122 cros_trunk FileSystemProviderApiTest.BigFile failure. Disabling the test, passing bug to owners.
                    • 457993 The pool is in a critical condition and cannot complete build verification tests in a timely manner.
                    Sheriffs: jwerner, victoryang, hungte
                    Gardener: girard
                    • reverted  - suspect it caused an x86 ASAN failure 
                    • 456993 Chrome PFQ failure
                    Sheriffs: dbasehore, katierh
                    Gardener: ihf
                    • 453090 Pre-CQ failure
                    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
                    • SSL connection flake for build
                    • 455728: ASAN unittest failure in permission broker
                    • 456501: canaries dying during ChromeSDK due to missing gbm.h header
                    • 456491: chrome pfq dying during BuildPackages due to dpkg-architecture errors
                    • 456829: arm-generic_freon chrome pfq failing with conflicting minigbm/mesa depends
                    Sheriffs: djkurtz
                    • 448208 - pool: bvt, board: daisy_spring in a critical state
                    • 454561 - pool: bvt, board: expresso in a critical state
                    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
                    Sheriffs: vbendeb, armansito, wuchengli
                    Gardener: jamescook
                    • 401341: update_engine UnitTest failures in P2PManagerTest.ShareFile, out of disk space on /tmp
                    Sheriffs: wuchengli
                    Gardener: jamescook
                    • 452349: Canary Chrome failures because of mixed Freon / non-Freon
                    • 36103: storm-release: BuildPackages failed in chromeos-base/ap-daemons
                    • 453201: [bvt-inline] provision Failure on zako-release/R42-6735.0.0
                    • 428058: [bvt-inline] security_NetworkListeners Failure on daisy_spring-chrome-pfq/R40-6412.0.0-rc2
                    • 446221: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures -> disabled
                    Sheriffs: sonnyrao, arakhov, vapier
                    Gardener: jamescook
                    • 452911: Chrome PFQ failing due to ozone/evdev/ warnings -> reverted, asked chromeos-tpms to bump PFQ
                    • 450335: [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq -> flaky test -> disabled
                    • 446221: cros_trunk: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures on official builders
                    • 452623: cros_trunk: WebRtcSimulcastBrowserTest.TestVgaReturnsTwoSimulcastStreams browser_tests failures -> disabled
                    • 453090: pre-cq failing with commit KeyError
                    • 453208: cidb connection failed with buildStageTable key error
                      Sheriffs: zeuthen, shawnn, vapier
                      Gardener: jamescook
                      • 452497: canaries all dying in chrome with /home/chrome-bot/depot_tools/external_bin errors
                      • 452534: pre-cq bots timing out due to most slaves offline
                      • 450278: Chromium OS Asan bots failing in logging_AsanCrash, telemetry exception problem
                      • 451603: Chromium OS (amd64) Asan: security_SandboxLinuxUnittests failing
                      • 449103: cros_trunk: WebInputEventAuraTest.TestMakeWebKeyboardEventWindowsKeyCode fails under ThreadSanitizer
                      • 371290: cros_trunk: ICOImageDecoderTest.Decoding content_unittest fails on 8010 Mac, Linux32, Linux64 bots
                      • 452647: cros_trunk builder failures: base_unittests: test.exe no such option --parallel
                      • 452706: syncing bluez repo broke with upstream ref errors
                      2015-01-23 - 2015-01-26
                      Sheriffs: zeuthen, shawnn, reveman
                      Gardener: tbarzic
                      • 452073: Beltino-B builder unable to build chrome from source.
                      • 452070: Missing prebuilts for nyan_freon.
                      • 452329: Chrome PFQ uprev failure.
                      2015-01-20 - 2015-01-21
                      Sheriffs: garnold, avakulenko, itspeter
                      Gardener: xiyuan, zork
                      • 445705: peach-pit ethernet issues cause update signals to not be received, failing autoupdate_EndToEnd.
                      • 450244: paygen timing out waiting on rambi-c-canary, waiting for DUTs.
                      • 450407: A CL in chryptohome seems to cause a unit test to fail. Reverted.
                      • 450771: Chrome PFQ is broken on MIPS platform. Related to this CL.
                      2015-01-14 - 2015-01-15
                      Sheriffs:  wfrichar, adlr, kpschoedel
                      Gardener: skuhne
                        • Network issues:
                        • veyron-pinky-nightly-chrome-pfq is red, looking at log seems a flake, rebuild
                        • Reverted since it broke many builders and updated the PFQ build to get the PFQ to uprev.

                        2015-01-12 - 2015-01-13
                        Sheriffs:  bfreed, bsimonnet, rongchang
                        Gardener: achuith
                        • Scheduled Lab shutdown on Jan 9 is complete.  Let's see if the tree comes back up.
                        • Chrome failed in the PFQ: git error.  Should not close our tree on PFQ failure.
                        • Tree throttled due to video_ChromeHWDecodeUsed Failure.
                        • beltino-freon full release failed to build binutils and chrome.  First build, so might be just plain broken.
                        • Canary timeouts in report stage, but jrbarnette did some additional cleanup as well.

                            2015-01-08 - 2015-01-09
                            Sheriffs:  quiche, bhthompson, rongchang
                            Gardener: achuith
                            • rojen and tkensinger replaced old winky duts with new ones

                                2015-01-06 - 2015-01-07
                                Sheriffs:  bhthompson, quiche, sheckylin, rongchang

                                  2015-01-06 - 2015-01-07
                                  Sheriffs:  grundler, jrbarnette, owenlin
                                  Gardener: oshima
                                  • 445705: AU and Paygen failures on peach_pit
                                    • Closed 446463 as a duplicate.
                                    • Two tickets filed:  b/18918701 b/18936609
                                    • No root cause:  expect more peach_pit failures to follow
                                  • CanaryCompletion timeouts caused by master restart (yjhong/cmasone)
                                  • winky DUTs in lab *locked* by rojen - caused winky paygen test failures
                                    • The DUTs were locked in order to replace them with MP hardware.
                                  • 322072: peach-canary, nyan-canary and winky timed out in paygen test
                                  • 446177: intermittent login test failures on x86, especially VM tests.
                                  • 446463: AU test failure on peach_pit.
                                  • 446885: security_OpenFDs failing in vmtests on asan bots
                                  • CL:239300: sync errors due to glibc upstream/ refs changing from a file to a dir

                                  2015-1-2 - 2015-1-5
                                  Sheriffs: benchan, namnguyen, dhendrix  Gardener:
                                  • 445068: logging_CrashServices found to be bricking DUTs, temporarily disabled
                                  • 286343: git push failures: missing permissions
                                  OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)