Sheriff Log: Chromium OS

2015-5-20
Sheriff:  owenlin, gwendal
    • 434201security_SandboxLinuxUnittests_SERVER_JOB Failure
    • CL:*216896: whirlwind canary failing signer tests due to new kernel option
    • 490057: hwlab burned down
    • Chrome PFQ failing on x86 bots with use_sysroot error -> already fixed in Chrome; waiting for new LKGM on Chrome side to include fix
    • 490546: ImageTest fails on mips bots due to unresolved symbols in mesa; CL:272365 reverted in the mean time
    • 491012: chrome pfq build failure in gbm_surface_factory.cc
    • 491103: samus paladin crashed while talking to cidb
    2015-5-19
    Sheriff:  tbroch, gwendal, jchuang
      • 483661x86-generic full: vmtest failed in SimpleTestUpdateAndVerify
      • 489733: veyron canary failed to build openssh.  Missing symbols from netdb ... part of glibc 
      • 472858: x86-generic ASAN, login_OwnershipApi:  fails when dbus "Did not receive a reply"
      • 477928: jecht-release-group: tidus: autoupdate_Rollback: unable to find host ... network flake?
      • 428345auron-b-release-group: lulu: platform_Powerwash,Host did not return from reboot
      2015-5-18
      Sheriff:  tbroch, gwendal, jchuang
      • 489302: peach-pits failing in lab.  Loose physical connection for entire rack in lab led to devices discharging.
      • 488644slippy-release-group: The AUTest [falco_li] [au] stage failed, 
      • 489215PFQ Master: binhosttest fail: AssertionError - cannot find Chrome prebuilts

        2015-5-15
        Sheriff:  itspeter, pstew, ejcaruso
        • 488539: oak canary was moved out of experimental and failed due to a large non-bisectable audio patch series going into 3.18
        • 488481: devserver fell over during a test, but it seems like a transient failure
        • 488580: image_to_vm failing; jrbarnette@ trying to find an owner
        • 488487488635: chrome crashes, achuith@ trying to find out how to roll back, pstew@ pinned chrome to 44.0.2401.3_rc-r1
        • 488644: ivybridge-freon failed during paygen step
        • 488739: bad X11 header include is breaking Chrome PFQ for freon images
        • 488959: buffet unittests failing on amd64 asan

        2015-5-14
        Sheriff: djkurtz, pstew, ejcaruso
          • 487955: veyron_pinky ChromeSDK fail - touch_handle_drawable_aura.cc - ui/resources/grit/ui_resources.h: No such file or directory
          • 487959: nyan_blaze BuildPackages fail - chromeos-base/chromeos-factory
          • 487990: All chromiumos.chromium builders broken by video_encode_accelerator_unittest
          • 487997chrome PFQ - BuildPackage fails - chromeos-chrome-44.0.2401.0_rc-r1 - keycode_text_conversion_ozone.cc - ui/events/keycodes/dom3/dom_code.h: No such file or directory
            • Suspect Chrome CL
          • 461087: Chrome PFQ fails with login_OwnershipNotRetaken HWTest failure on Tricky -- sheriffs get email on this with strong verbiage that Chrome OS won't be picking up a new Chrome for this reason, but it turns out that the PFQ master ignores this build failure because Tricky is experimental.
          • b/21165021: Canaries fail if they tried to connect to databases/servers between 2:45pm and 3:00pm PST due to power brownout
          • 488291: x86-generic ASAN failure in login_LoginSuccess
          • 488308: various paladin builders failing due to infrastructure issues

          2015-5-12/13

          Sheriff: wiley, puneetster

          • 460860: login_Cryptohome Failure on tricky_freon-release
          • 487120squawks boards in short supply in BVT pools, hwtests won't finish in time
          • 487303: 503 killed suite job during Leon canary
          • 433204: autoupdate_EndToEndTest.paygen_au_canary_test_full fails "... no attribute 'Error'"
          • 487772: amd64-generic ASAN builder has been red since April 6?!?

          2015-5-10/11
          Sheriff: cychiang

          • 486497: paygen failing: "Payload integrity check failed: install_operations[615](REPLACE_BZ): src_length is zero."  Revert CL
          • 465711: Host did not return from reboot. This also breaks PFQ
          • 487055: stout canary: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **
          • 487123: Canary builders got killed by signal 9 around May 11 22:58:31 2015 buildbot time.



          2015-05-04/05
          Sheriffs: garnold, jwerner
          Gardener: stevenjb
          • 476649: Sandybridge canary failure due to lumpy DUT connectivity issue.
          • b/20859395: Stumpy moblab canary failed on hwtest.
          • b/20859821: Daisy canary failed on hwtest.
          • Reverted: https://codereview.chromium.org/1121293002 (suspected causing flaky interactive_ui_tests failure)
          • 482446: Sporadic failures
          • 477889: Sporadic failures
          • 484307: Still in-progress. Much time spent trying to track this down.
          • 484726: Multiple canary failures due to lab DNS flake.
          • CL:268999: Stout vmtest failing due to bad CL.
          • 484243: Multiple canary failures due to lab test fail on ACL error.
          • 486497: most canaries dying in paygen with "install_operations[1219](REPLACE_BZ): src_length is zero."

          2015-05-01
          Sheriffs: posciak
          • 476550 ivb failures on TestBlacklistedFileTypes, TestValidInterpreter, etc.
          • 482905 slippy ABORT: reboot command failed
          • 477739 samus suite prep abort

          2015-04-29
          Sheriffs: snanda, benzh, hungte
          • 483174: setuid bit missing on ping, breaking ping in chroot
          • 476649Some lumpy duts unreachable
          • 482956PFQ Master: AssertionError - cannot find Chrome prebuilts
          • 482905: provision: ABORT: Host did not return from reboot
          • 433389: [Samus] Kernel crash meta files shown after rebooting device
          • 463861: [bvt-inline] provision Failure on lumpy-release/R42-6812.15.0
          • 482454: Delta to itself failed with NewRootfsVerificationError in veyron_speedy-release/R44-7019.0.
          • CL:*215145 storm premp rejected by signer due to security checks forced off
          2015-04-28
          Gardener: afakhry
          • 482121 TabindexSaveFileDialog/FileManagerBrowserTest.Test/2 has been failing on the official cros trunk.
          • 409019 graphics_GpuReset Failure on falco PFQ.
          • 482171 graphics_Sanity_SERVER_JOB Failure on falco PFQ.
          • 482164 login_LoginSuccess Failure on lumpy PFQ.
          • 481732 Flakey provision failure on falco PFQ.
          • 482454 veyron_speedy failing AU test
          • CL:*215145 storm premp rejected by signer due to security checks forced off
          2015-04-27
          Gardener: afakhry
          • 481544 Chrome PFQ is failing on x86-generic, alex, lumpy, peppy and tricky due to desktopui_ScreenLocker VMTest failures.
          • 481820 bluez unittests breaking on amd64-generic
          • 481864 asan bots failing unittests: libchromeos-streams-323904.so: error adding symbols: DSO missing from command line
          • 481872 amd64 asan bot failing unittest due to leaks in libchromeos
          2015-04-24
          Gardener: derat
          Sheriffs: posciak
          • 466719 lumpy PFQ failures "Unhandled AutoservSSHTimeout"
          • 465178 tricky PFQ failed 3 times due to a login timeout in telemetry_LoginTest, but then passed again
          • 476584 (possibly) samus-canary failure, issues in TestBlacklistedFileTypes
          • More ASAN failures, looks like 478605 not fixed after all?
          • 480638 MenuControllerMnemonicTestMnemonicMatch.MnemonicMatch failing on cros trunk
          • 480667 SSLUITest.BadCertFollowedByGoodCert failing on cros trunk
          • 470130 Chromium OS (x86) Asan builder still always failing with MySQL error (one month and counting!) 
          2015-04-23
          Gardener: derat
          Sheriffs: moch, wuchengli
            • 463805 PFQ lumpy run failed with "AutoservSSHTimeout: ('ssh timed out', * Command: )"
            • 480491 TabindexOpenDialog_FileManagerBrowserTest browser_tests failing on cros trunk
            • 470130 Chromium OS (x86) Asan builder failing with "Can't connect to MySQL server on '173.194.81.53' (110)"
            • 478605 Pretty much all tests failing on Chromium OS (amd64) Asan builder (at least, I hope this is the cause)
            • 480514 lxc container failed to get IP address from bridge
            • 480638 MenuControllerMnemonicTestMnemonicMatch.MnemonicMatch failing on cros trunk
            • 480667 SSLUITest.BadCertFollowedByGoodCert failing on cros trunk
            • 465862 desktopui_ScreenLocker failing on Chromium OS (amd64) Asan
            • 449361 HWTest did not complete due to infrastructure issues (code 3)" on enguarde
            2015-04-22
            Sheriffs: moch, tyanh
            2015-04-20, 2015-04-21
            Sheriffs: dbasehore, alecaberg, tyanh
            • 476550 paygen issue for signing images
            • 477739 bvt suite abort
            • 463805 Provision failure on SSH timeouts
            • 478713 Autotest execution errors
            • 478605 CrOs asan failures
            • 478762 mod-image-for-recovery failure on non-x86
            2015-04-17
            Sheriffs: semenzato, quiche, mcchou (shadow)
            • 477747 ImageTest failure on canries due to /usr/include/tiffio.hxx, /usr/libexec/perf-core/tests/attr.py, and /lib64/libthread_db-1.0.so
            • 477883 kayle-paladin failed due to chromeos-initramfs
            • 477888 CQ failure due to VM test timeout in buffet_InvalidCredentials
            • 477889 CQ failure due to VM test timeout in buffet_Registration
            • 477941 link pre-cq failure (with no CLs): build_image failed due to no space on loop device
            • 473970 canary failure due to autoupdate_EndToEndTest
            • 474831 moblab_runsuite failure: /root/.boto did does not exist
            • 478605 asan bots both dead in vmtest w/new chrome
            • CL:266444 non-x86 archive failures in mod_image_for_recovery
            2015-04-16
            Sheriffs: semenzato, quiche, mcchou (shadow)
            • 477703 make_factory_toolkit.sh broke Archive step
            • 477712 enabling frame pointers broke optimized webrtc code in chrome
            • 477739 samus-release failed in suite prep
            2015-04-15
            Sheriffs: avakulenko, jrbarnette, fjhenigman
            • 476434 Network flakiness causes a dozen canaries go red.
              • As of this evening, the problem is still causing intermittent failures across the board.
            • 477352 Chrome fails to start on jerry and mighty
              • Early morning: Chrome has been pinned to 44.0.2368.0 until the problem is fixed.
              • Mid-day: Testing showed the problem is on the Chrome OS side; reverted this CL.
              • Afternoon: Chrome is unpinned.
              • Evening: Waiting for the CL to make it through a canary, so that the veyron builders will show green.
            2015-04-14
            Gardener: jonross
            Sheriffs: avakulenko, jrbarnette, kpschoedel
            • 438729 OutOfProcessPPAPITest.MediaStreamAudioTrack flaky on CrOS. Taking out the X server, causing other tests to fail.
            • 469119 TouchExplorationTest.RewritesEventsWhenOn is flaky
            • 476934 Chrome PFQ bots failing with config failures, blocking uprev.
            • 475923 TouchExplorationTest.SplitTapExplore flaking 50% of the time.
            • 476550 Paygen signing failure on multiple boards (including veyron)

            2015-04-10 and  2015-04-13
            Gardener: jonross
            Sheriffs: kpschoedel, denniskempin, bleung
            • 476607 Peach_pit Chrome PFQ failng HWTest step due to Infra issue.
            • 476577 KioskUpdateTests are flaky on Linux Chromium OS.
            • 469119 Flaky TouchExplorationTest.RewritesEventsWhenOn is causing failures on CrOS trunk
            • 476338 PFQ Failure on lumpy. Provision failure tracking SSH timeout.
            • 476434 Tree is throttled, devservers are causing a P0 issues causing AU testing to flake heavily for the morning PST
            2015-04-08
            Gardener: zork
            • 475170: Autotest security_OpenFDs failing.

            2015-04-06 and 2015-04-07
            Gardener: xiyuan
            Sheriffs: deymo, shawnn, vapier, deanliao
            • 474227: Lack of speedy DUTs in lab
            • 465862: desktopui_ScreenLocker failing on asan bots
            • br:764: buffet autotests flaking on amd64-generic-full
            • 474497: Canaries dead in paygen
            2015-04-02 and 2015-04-03
            Gardener:
            Sheriffs: wfrichar, adlr, itspeter (on holiday), vapier
            • cbuildbot gsutil errors, solved by fixing permissions manually (by system services team) and https://chromium-review.googlesource.com/#/c/263633/1
            • Unittest break in germ, quickl fixed by Jorge
            • repo upload failing on kernel repo: b/20062832b/19932429
            • CL:263970 gnutls pulled due to new pyshark code (CL:262457)
            • 473738: sumo board failing to build adhd
            • 473742: rush_ryu recovery kernel failing
            • br:344: kernel git checkouts failing on gizmo/project-sdk bots
            • CL:*211751: nyan_freon failing signer tests
            • 473721: paygen ran out of space on swanky
            • 473899: paygen not finding images
            • 473900: mips qemu fc-cache call crashing

            2015-03-31 and 2015-04-01
            Gardener: 
            Sheriffs: bfreed, bhthompson, josephsih
            • Throttled tree in the morning, most canaries red.  471656: Paygen: failed to set up loop device
            • 472378: x86 asan builder failing unittests due to leak san not being supported
            • 472658: Paygen fails: Permission denied: '/dev/loop27p4'
              • The reverts above get the canaries past "failed to set up lop device", but many now fail with "Permission denied: '/dev/loop27p4'".
            • 449738: "Internal server error" should be treated as infrastructure failure so it doesn't reject CLs
              • "CQ encountered a critical failure" email.
              • Pointer to https://uberchromegw.corp.google.com/i/chromeos/builders/CQ%20master/builds/5150 shows most failures due to "FAILED RPC CALL: get_jobs" in HWTest.
            • 347423: Gerrit failed to submit change.
              • Sounds like a transient failure.  Should just try it again.
              • 13:57:51: WARNING: Change 263241 was submitted to gerrit without errors, but gerrit is reporting it with status "NEW" (expected "MERGED").
              • 13:57:51: ERROR: Most likely gerrit was unable to merge change 263241.
            • 472858: VmTest fails when dbus "Did not receive a reply"

            2015-03-30 and 2015-03-31
            Gardener: 
            Sheriffs: josephsih
            • 471656: Paygen: failed to set up loop device: No such file or directoryIs. This is a new issue causing the failure of 20 canary builders.
            • 466777: HW_Test error: this issues still exists.
            • 469259: image_to_vm.sh fails: this issue still exists.

            2015-03-26 and 2015-03-27
            Gardener: skuhne
            Sheriffs: puthik, furquan
            • PFQ failed to uprev. Caused by 991533002, reverted (Chrome and ChromeOS PFQ).
            • 469566: pool: bvt, board: veyron_jerry in a critical state
            • 466777: pool: bvt, board: veyron_speedy in a critical state
            • 469259: image_to_vm.sh fails with "partx: /dev/loop0: error deleting partition ..."
            • 471032: PFQ still fails - now: The binary size of the image exceeds the limits. (e.g. Daisy-freon, Daisy-skate)
            • 449766PFQ still fails - now: [bvt-inline] login_LogoutProcessCleanup Failure on Falco

            2015-03-24 and 2015-03-25
            Gardener: jamescook
            Sheriffs: charliemooney, kcwu
            • 463530461087  Chrome uprev failure on tricky, cryptohome not mounted
            • 466719 lumpy ssh timeout resulting in provisioning failures
            • 470172 MasterUploadPrebuilts failure in Update PFQ config dump
            • 470118 Amd64-generic failing vmtest "buffet_Registration" with urlopen "Connection Refused"
            • 470237: WallpaperManagerBrowserTest.DevLaunchApp failing on cros_trunk official builders
            • 470130: Chromium OS (x86) Asan always failing with MySQL connection error
            • 470381: [bvt-cq] graphics_SanAngeles Failure on tricky-chrome-pfq/R43-6910.0.0-rc5 - wflinfo / waffle problem, test disabled
            • 1035763003: Revert of Test Accelerators In Interactive UI Tests - cros_trunk official builder was failing
            • 448247: [bvt-inline] provision Failure on candy-release/R42-6683.0.0 - update_engine failed
            • 470701: Flaky BVT security_Firewall failure, "Mismatched firewall rules"
            2015-03-18 and 2015-03-19
            Gardener: abodenha
            Sheriffs: stevefung
            • 468340, 468770: PFQ and canaries are a mess due to infra issues
            • 465862: amd64 ASAN builds are failing
            • 468394: autoupdate_EndToEndTest.paygen_au_stable_test_delta failures
            • 466777: not enough duts
            • 467975: Image Signing Timeouts
            • 463805:  autoserv timeouts
            2015-03-17
            Sheriffs: littlecvr,stevefung

            • 467975: Image Signing Timeouts
            • 260602: Fix auron_paine build break

            2015-03-16
            Sheriffs: littlecvr,dlaurie,zqiu

            • 466972: insufficient DUTs for butterfly
            • 466777: insufficient DUTs for veyron_speedy
            • 465230: wolf: login_OwnershipTaken_SERVER_JOB Failure
            • 433970: wolf: login_LogoutProcessCleanup_SERVER_JOB Failure
            • 419772: gnawty: security_ProfilePermissions_SERVER_JOB Failure
            • 411608: gnawty: security_NetworkListeners_SERVER_JOB Failure
            • 434148: gnawty: login_MultiUserPolicy_SERVER_JOB Failure


            2015-03-13
            Sheriffs: tyanh,dlaurie,zqiu

            • 426164: [au] autoupdate_EndToEndTest.npo_test_delta Failure on nyan_blaze-release/R38-6158.71.0
            • 466919: [paygen_au_canary] autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on quawks-release/R43-6872.0.0
            • 8 issues on [bvt-inline] login_* Failure on wolf-release/R43-6872.0.0
              • 434202: login_RetrieveActiveSessions_SERVER_JOB Failure
              • 434182: login_SameSessionTwice_SERVER_JOB Failure
              • 434178: login_OwnershipNotRetaken_SERVER_JOB Failure
              • 434185: login_Cryptohome_SERVER_JOB Failure
              • 419772: security_ProfilePermissions_SERVER_JOB Failure
              • 434148: login_MultiUserPolicy_SERVER_JOB Failure
              • 403701: login_MultipleSessions_SERVER_JOB Failure
              • 434195: login_GuestAndActualSession_SERVER_JOB Failure
            • b/19729024: incorrect DHCP config change was pushed breaking DHCP in the lab
            • br/590: amd64 asan failing unittests due to leaks related to protobuf

            2015-03-12
            Sheriffs: tyanh

            • 462734: [sanity] provision Failure on lumpy-release/R43-6869.0.0

            2015-03-10
            Sheriffs: chirantan, thieule
            Gardener: oshima

            • 465752: [bvt-inline] login_RetrieveActiveSessions Failure on tricky
            • 465963: daisy canary: The AUTest [daisy_spring] [au] stage failed: ** HWTest did not complete due to infrastructure issues (code 3)
            • 464171: Multiple canary's are failing due to kernel size limits
            • 465877: [bvt-inline] login_OwnershipTaken Failure on x86-mario-release/R43-6865.0.0
            • 464751: [au] provision Failure
            2015-03-09
            Sheriffs: chirantan, thieule
            Gardener: achuith

            • 464938: Test lab performance issue
            • 465596: update_engine is failing on all canary builders
            2015-03-06
            Sheriffs: benchan, tbroch
            Gardener: tengs


            2015-03-05
            Sheriffs: benchan, tbroch
            Gardener: tengs
            • 464407 Your "Oauth 2.0 User Account" credentials are invalid .... Failure: Invalid response 302..

            2015-03-04
            Sheriffs: dhendrix, waihong
            Gardener: derat

            2015-03-03
            Sheriffs: dhendrix, waihong
            Gardener: derat
            • 462842 Chromium OS (amd64) Asan bot failing unittests due to webserver leaks
            • 463493 Flaky failure in breakpad's linux_client_unittest on X86 (chromium)
            • 463532 browser_tests failing on cros trunk in WebViewTest.FileSystemAPIRequestFromWorkerDeny
            • 464053 Beaglebone kernel too big, causing build failures
            • 463411 pool: bvt, board: leon in a critical state. (Some DUTs had their USB ethernet dongles swapped and were down for a bit)
            2015-03-02
            Sheriffs: dgreid, tbroch
            Gardener: derat
            • 461406 Chromium OS (x86) Asan bot still failing
            • 458775 peach_pit nightly chrome PFQ failed with too few available DUTs
            • 463213 X86 (chromium), Daisy (chromium), and AMD64 (chromium) failing on chromeos-chrome with "ValueError: invalid literal for int() with base 10"
            2015-02-27
            Sheriffs: dgreid, tbroch
            2015-02-25
            Sheriff: dianders, pstew, jchuang
            • 462240 [storm-release canary] storm-release: The BuildPackages stage failed: Packages failed in ./build_packages
            • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
            • 461841 rambi-c-release-group: timed out

            2015-02-25
            Sheriff: dianders, pstew, tyanh
            • 461184 [canary] HWTest did not complete due to infrastructure issues again in canary 670; also "peach-release-group: timed out"
            • 461841 [canary] sandybridge-release-group: The Paygen [butterfly] stage failed: <class 'chromite.lib.timeout_util.TimeoutError'>: Timeout occurred- waited 13800 seconds
            • 461893 [canary] rambi-a-release-group: The HWTest [expresso] [bvt-inline] stage failed: ** Suite timed out before completion **
            • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
            2015-02-24
            Sheriff: amstan, gwendal, tyanh
            • 461184 [canary, chrome pfq] HWTest did not complete due to infrastructure issues (HWTest Lab closed?)
            • 461188 [pineview canary] Operation timed out at the end of a build; transient, subsequent build succeeded.
            • 438908 Image signing timed out across many platforms on canary; transient, subsequent builds succeeded.
            • 461378 chrome compilation error prevents some build to complete in CQ.
            • 461415 critical fix in chrome was only present on chrome ToT.

            2015-02-23
            Sheriff: amstan, gwendal  Gardener: flackr
            • 460815 Cros SDK broken in .bashrc_profile
            • 460693 Chrome PFQ failing with exception: global name 'AccessDeniedException' is not defined in File "src/build/download_sdk_extras.py", line 71
            • 460951 ChromeMetricsServiceAccessorTest.MetricsReportingEnabled, ExternalCacheTest.Basic, ExternalCacheTest.PreserveInstalled, DeviceLocalAccountExternalPolicyLoaderTest.ForceInstallListSet failing on cros_trunk
            • 458122 Reopened FileSystemProviderApiTest.BigFile failing on cros_trunk as test is failing 98% of runs on cros_trunk
            • 461021 MetricsServicesManagerTest.GetRecordingLevelCoarse faililng on cros_trunk.
            • 461046 MetricsServicesManagerTest.GetRecordingLevelFine faililng on cros_trunk.
              2015-02-18
              Sheriff: posciak, namnguyen, snada
              • Note to PST sheriffs: lots of infrastructure failures (I think) over the last few days, which I don't really understand. I think we need help from Infra team before we can reopen the tree for good.
              • 419904 IndexError: list index out of range on moblab_RunSuite 
              • 458613 Pre-CQ Launcher failures
              • 224 Moblob failures
              • 459679: Moblab blocks canaries
              • b/19426205: Missing commit info

              2015-02-13
              Gardener: jonross
              • Reverted a change that was failing a compile on Chrome OS.
              • 458567 Disable flaky InProcessAccessibilityBrowserTest.VerifyAccessibilityPass
              • 458549 Disable flaky TextInput_TextInputStateChangedTest.SwitchingAllTextInputTest
              • 458526 Chromium OS Waterfall bots falling on LKGMSync
              • 458918 amd64 asan failing shill unittests due to leaks
              2015-02-12
              Gardener: jonross
              • 458341 Disable flaky LoginPromptBrowserTest.LoginInterstitialShouldReplaceExistingInterstitial
              • 458333 Disable flaky AutofillDialogControllerSecurityTest.DoesntWorkOnBrokenHttps
              • Reverted patch that broke Chrome OS LoginUITests
              • 458154 Chrome PFQ peach_pit. HWTest failure, RPC Connection Timeout.
              • 458122 cros_trunk FileSystemProviderApiTest.BigFile failure. Disabling the test, passing bug to owners.
              • 457993 The pool is in a critical condition and cannot complete build verification tests in a timely manner.
              2015-02-10
              Sheriffs: jwerner, victoryang, hungte
              Gardener: girard
              • reverted https://codereview.chromium.org/899973006  - suspect it caused an x86 ASAN failure 
              • 456993 Chrome PFQ failure
              2015-02-03
              Sheriffs: dbasehore, katierh
              Gardener: ihf
              • 453090 Pre-CQ failure
              • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
              • SSL connection flake for build
              • 455728: ASAN unittest failure in permission broker
              • 456501: canaries dying during ChromeSDK due to missing gbm.h header
              • 456491: chrome pfq dying during BuildPackages due to dpkg-architecture errors
              • 456829: arm-generic_freon chrome pfq failing with conflicting minigbm/mesa depends
              2015-02-02
              Sheriffs: djkurtz
              Gardener: 
              • 448208 - pool: bvt, board: daisy_spring in a critical state
              • 454561 - pool: bvt, board: expresso in a critical state
              • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
              2015-01-30
              Sheriffs: vbendeb, armansito, wuchengli
              Gardener: jamescook
              • 401341: update_engine UnitTest failures in P2PManagerTest.ShareFile, out of disk space on /tmp
              2015-01-29
              Sheriffs: wuchengli
              Gardener: jamescook
              • 452349: Canary Chrome failures because of mixed Freon / non-Freon
              • 36103: storm-release: BuildPackages failed in chromeos-base/ap-daemons
              • 453201: [bvt-inline] provision Failure on zako-release/R42-6735.0.0
              • 428058: [bvt-inline] security_NetworkListeners Failure on daisy_spring-chrome-pfq/R40-6412.0.0-rc2
              • 446221: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures -> disabled
              2015-01-28
              Sheriffs: sonnyrao, arakhov, vapier
              Gardener: jamescook
              • 452911: Chrome PFQ failing due to ozone/evdev/input_controller_evdev.cc warnings -> reverted, asked chromeos-tpms to bump PFQ
              • 450335: [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq -> flaky test -> disabled
              • 446221: cros_trunk: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures on official builders
              • 452623: cros_trunk: WebRtcSimulcastBrowserTest.TestVgaReturnsTwoSimulcastStreams browser_tests failures -> disabled
              • 453090: pre-cq failing with commit KeyError
              • 453208: cidb connection failed with buildStageTable key error
                2015-01-27
                Sheriffs: zeuthen, shawnn, vapier
                Gardener: jamescook
                • 452497: canaries all dying in chrome with /home/chrome-bot/depot_tools/external_bin errors
                • 452534: pre-cq bots timing out due to most slaves offline
                • 450278: Chromium OS Asan bots failing in logging_AsanCrash, telemetry exception problem
                • 451603: Chromium OS (amd64) Asan: security_SandboxLinuxUnittests failing
                • 449103: cros_trunk: WebInputEventAuraTest.TestMakeWebKeyboardEventWindowsKeyCode fails under ThreadSanitizer
                • 371290: cros_trunk: ICOImageDecoderTest.Decoding content_unittest fails on 8010 Mac, Linux32, Linux64 bots
                • 452647: cros_trunk builder failures: base_unittests: runtest.py test.exe no such option --parallel
                • 452706: syncing bluez repo broke with upstream ref errors
                2015-01-23 - 2015-01-26
                Sheriffs: zeuthen, shawnn, reveman
                Gardener: tbarzic
                • 452073: Beltino-B builder unable to build chrome from source.
                • 452070: Missing prebuilts for nyan_freon.
                • 452329: Chrome PFQ uprev failure.
                2015-01-20 - 2015-01-21
                Sheriffs: garnold, avakulenko, itspeter
                Gardener: xiyuan, zork
                • 445705: peach-pit ethernet issues cause update signals to not be received, failing autoupdate_EndToEnd.
                • 450244: paygen timing out waiting on rambi-c-canary, waiting for DUTs.
                • 450407: A CL in chryptohome seems to cause a unit test to fail. Reverted.
                • 450771: Chrome PFQ is broken on MIPS platform. Related to this CL.
                2015-01-14 - 2015-01-15
                Sheriffs:  wfrichar, adlr, kpschoedel
                Gardener: skuhne
                  • Network issues: https://b2.corp.google.com/u/0/issues/19028546
                  • veyron-pinky-nightly-chrome-pfq is red, looking at log seems a flake, rebuild
                  • Reverted https://codereview.chromium.org/857613002/ since it broke many builders and updated the PFQ build to get the PFQ to uprev.

                  2015-01-12 - 2015-01-13
                  Sheriffs:  bfreed, bsimonnet, rongchang
                  Gardener: achuith
                  • crbug.com/447821: Scheduled Lab shutdown on Jan 9 is complete.  Let's see if the tree comes back up.
                  • crbug.com/448079: Chrome failed in the PFQ: git error.  Should not close our tree on PFQ failure.
                  • crbug.com/446889: Tree throttled due to video_ChromeHWDecodeUsed Failure.
                  • crbug.com/448244: beltino-freon full release failed to build binutils and chrome.  First build, so might be just plain broken.
                  • crbug.com/448414: Canary timeouts in report stage, but jrbarnette did some additional cleanup as well.

                      2015-01-08 - 2015-01-09
                      Sheriffs:  quiche, bhthompson, rongchang
                      Gardener: achuith
                      • crbug.com/447324: rojen and tkensinger replaced old winky duts with new ones

                          2015-01-06 - 2015-01-07
                          Sheriffs:  bhthompson, quiche, sheckylin, rongchang
                          Gardener: 

                            2015-01-06 - 2015-01-07
                            Sheriffs:  grundler, jrbarnette, owenlin
                            Gardener: oshima
                            • 445705: AU and Paygen failures on peach_pit
                              • Closed 446463 as a duplicate.
                              • Two tickets filed:  b/18918701 b/18936609
                              • No root cause:  expect more peach_pit failures to follow
                            • CanaryCompletion timeouts caused by master restart (yjhong/cmasone)
                            • winky DUTs in lab *locked* by rojen - caused winky paygen test failures
                              • The DUTs were locked in order to replace them with MP hardware.
                            • 322072: peach-canary, nyan-canary and winky timed out in paygen test
                            • 446177: intermittent login test failures on x86, especially VM tests.
                            • 446463: AU test failure on peach_pit.
                            • 446885: security_OpenFDs failing in vmtests on asan bots
                            • CL:239300: sync errors due to glibc upstream/ refs changing from a file to a dir

                            2015-1-2 - 2015-1-5
                            Sheriffs: benchan, namnguyen, dhendrix  Gardener:
                            • 445068: logging_CrashServices found to be bricking DUTs, temporarily disabled
                            • 286343: git push failures: missing permissions
                            OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)
                            Comments