Sheriff Log: Chromium OS

CrOS gardener:
Sheriff: tfiga
  • 539720Waterfall can't accessed through Uberproxy, need to use
  • x86-generic full VMTest login_Cryptohome flake

2015-10-06 and 2015-10-05
CrOS gardener: stevenjb
Sheriff: bsimonnet, furquan
  • 539594: depthcharge build failure on veyron rialto [Fixed]
  • 539748: chromeos-chrome fails to build: rmdir/mkdir access violation in (all canaries failing) [Fixed]
  • 539720Waterfall can't accessed through Uberproxy, need to use
  • 539739: Flaky VMTest Failures

CrOS gardener:
Sheriff: tfiga
  • 539720Waterfall can't accessed through Uberproxy, need to use
  • 539748: chromeos-chrome fails to build: rmdir/mkdir access violation in (all canaries failing)
  • 539594: depthcharge build failure on veyron rialto still not fixed
  • 538908: still flakey

CrOS gardener: ihf
Sheriff: bowgotsai
  • 418539: Change to fixsecurity_NetworkListeners Failure is out for review.
  • 485108: Moved desktopui_FlashSanityCheck from bvt-cq to bvt-perbuild.
  • 538908: Lab should have recovered enough daisy_skate to continue testing.

CrOS gardener: ihf
Sheriff: bowgotsai
  • 537655: 'Platform' object has no attribute 'SetHTTPServerDirectories'
  • 535374: pre-cq failure: GOBError: Forbidden
  • 538017: paygen failure (autoupdate_EndToEndTest.paygen_au_dev_delta Failure)
  • 518591: provision Failure (DownloaderException: Could not find autotest.tar in Google Storage)
  • 505744: provision Failure (sanity,provision,Unhandled AutoservSSHTimeout)
  • 526453: autoupdate_Rollback Failure, ANCHOR TestFailure(au,autoupdate_Rollback,update-engine failed on chromeos4-row1-rack4-host3)
  • 538480: autoupdate_EndToEndTest.paygen_au_canary_full
  • 538476: autoupdate_EndToEndTest.paygen_au_canary_delta
CrOS gardener:
Sheriff: josephsih
  • 537886: paygen failure (Failed to finish download from devserver)
  • 530498HWTest failure (stage_artifacts timed out)
  • 485881HWTest failure (login_OwnershipNotRetaken)
  • 529466AUTest failure (Suite job failed or provisioning failed)
  • 534437: amd64-generic full failed on results-40-buffet_RestartWhenRegistered
  • 538098: Sync buildbot slave files failed: update_scripts failed
  • 538057ERROR: Failed to stage payload: stage_artifacts timed out
  • 538140: CQ HWTests failing in audio_CrasSanity: Unhandled AttributeError: 'Platform' object has no attribute 'SetHTTPServerDirectories'

CrOS gardener:
Sheriff: josephsih
  • 537419: autoupdate_EndToEndTest.paygen_au_dev__full: Unhandled KeyError: 'source_payload_uri'
  • 488291: amd64-generic ASAN failed on login_LoginSuccess
  • 534437: amd64-generic full failed on results-40-buffet_RestartWhenRegistered
  • 537799: PDFTestFiles/PDFExtensionTest.Load/2 and 8 are currently flaky on cros_trunk.

CrOS gardener:
Sheriff: kitching
  • 537075: falco-full-compile paladin fails due to missing sqlalchemy import
  • CL 301593: libgcrypt update causing problems with mesa-img
  • 537087: lakitu_next-incremental always failing VMTest security_Minijail0 and security_SuidBinaries
  • 409019: graphics_GpuReset failure (GPU hang) on falco-chrome-pfq
  • 536780: added more timeouts (stage_artifacts)
  • 536259: HW Lab Infrastructure: too few guado bvt boards (only 2, require 4)
  • 536794: HW Lab Infrastructure: too few stumpy bvt boards (only 3, require 4)
  • 537128: 8+ canary build trees failing on Autotest timeouts

CrOS gardener:
Sheriff: kitching
  • 536670autoupdate_EndToEndTest.paygen_au_* test failure: Canary still has tons of argument mismatch errors even after garnold reverted his CL
  • 536690: amd64-generic incremental build failure: updates to buffet package
  • CL 302593: fio update fails HWTest
  • 536515: HW Lab Infrastructure: too few auron_paine bvt boards (only 3, require 4)
  • 536618: HW Lab Infrastructure: too few tidus bvt boards (only 3, require 4)
  • 536775: Autotest: many builds failing due to security_ and login_ tests
  • 536780: HWLab: builds failing due to timeouts

CrOS gardener:
Sheriff: deanliao

Builder x86-generic full #17275
  • 472858 VMTest fail: DBusException: org.freedesktop.DBus.Error.NoReply
Several autoupdate_EndToEndTest.paygen_au_* test failures:
HWTest lost MySQL connection: 535795

CrOS gardener:
Sheriff: zachrmarcheu, itspeter

Mostly the same as 09-22. Things different from yesterday:
The following two CLs suspects to be root cause for Builder CQ master Build #7757 , 'NoneType' object has no attribute 'splitlines' .
fdeng@ had Verify-1 for those 2 CLs:
Chump-in CLs (Doesn't break anything, just FYI)

CrOS gardener: bruthig
Sheriff: briannorris, mylesgw, itspeter
  • Several infra failures point to 534361; fdeng is looking at it
Builder Canary master Build #1307
  • Builder jecht group canary Build #777
    • 491290 autoupdate_Rollback failures on rikku
  • Builder pineview group canary Build #1682
    • autoupdate_EndToEndTest dashboard shows alex is 80%. Give it a retry.
  • Builder auron-b group canary Build #847, Failure since 2015-09-19
    • 533881 [paygen_au_canary] provision Failure on gandof-release/R47-7472.0.0
    • 533879 [paygen_au_dev] provision Failure on gandof-release/R47-7472.0.0
Builder x86-generic full #15418
  • 532658 VMTest fail: DBusException: org.freedesktop.DBus.Error.NoReply

CrOS gardener: bruthig

  • 534437 vm_disk and vm_memory failed_SimpleTestVerify_1_autotest_tests on amd64-generic
  • 496555 [bvt-cq] provision Failure on falco-chrome-pfq/R45-7139.0.0-rc2
    • failing on daisy_skate
  • 534451 PFQ builds timing out when calling /b/cbuild/internal_master/chromite/third_party/swarming.client/
    • failed on tricky and falco
  • 530498 [sanity] provision Failure on tricky-chrome-pfq/R47-7446.0.0-rc3
  • 496292 [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq/R45-7137.0.0-rc2
    • failed on lumpy
  • 533979 [bvt-cq] audio_CrasSanity Failure on lumpy-chrome-pfq/R47-7474.0.0-rc1
  • 478533 [bvt-cq] desktopui_FlashSanityCheck Failure on lumpy-chrome-pfq/R44-6989.0.0-rc1
  • 418539 [bvt-inline] security_NetworkListeners Failure on daisy_skate-release/R39-6315.0.0
  • 534544 [bvt-cq] network_DefaultProfileServices Failure on tricky-chrome-pfq/R47-7478.0.0-rc2
  • 534399 IncidentReportingServiceTests are crashing on the cros_trunk build
  • 510291  DevToolsPixelOutputTests.TestScreenshotRecording fails on cros_trunk official bot

Sheriffs: posciak, cernekee, smbarber
  • 530203 Samus with on-going HWTest Aborts.
  • 491290 autoupdate_Rollback failures on beltino
  • 483749 more provision failures (daisy_skate)
  • 409019 graphics_GpuReset failures on lumpy
  • 533006530693 multi-GB log files from Link FSI (?) took down the lab network
  • 488291 x86-generic, amd64-generic ASAN vmtests are failing
  • omg/1030 unable to fetch binaries from GS

CrOS gardener: jonross
  • 530646 WebViewTest.Dialog_TestConfirmDialogDefaultGCCancel Failure from V8 Roll (Also 530593)
  • 530612 Canary master: BranchUtil: No such file or directory: '/tmp/cbuildbot-tmpd0s19p/tmprZ63D7/src/aosp/system/attestation'
  • 518591 Veyron-group provision failure.
  • 528748 Mario/nyan pool-health bug
  • 530203 Samus with on-going HWTest Aborts.
  • Actually passed last night! First pass since August 27th :D
  • 530498 Provision failure on Tricky
  • 471531 Peach pit failure on provision_AutoUpdate.
  • 530605 Falco PFQ Aborting during HWTest
  • 473976 Daisy_skate failure in HWTest
  • 530661 Lumpy audio_CrasSanity Failure

CrOS gardener: jonross
Sheriffs: drinkcat, snanda, tbroch

  • 530265 chromiumos sdk :: *-client pkgs need dep to chromeos-dbus-bindings.
  • 530194 Alex and Lumpy bots failing update_scripts on missing gclient config.
  • 529216 paygen failures now also appearing on Nyan, Rambi, and Veyron group
  • 528748 x86-mario HWTest pool issue.
  • 530203 Samus HWTest reported an infra issue.
  • 471531 Daisy AutoUpdate Failure. Looks to be a flake, hopefully clears up.
  • Other failures (Tricky, Falco) appears to be possible HWTest [bvt-inline] flakes, as the jobs timed out. No detailed logs or auto-filed bugs from failures.
  • 530286 Falco Failure, desktopui_ExitOnSupervisedUserCrash Failure: no process matching chrome
  • 489106 Lumpy Autotest failure during graphics_CpuReset. Unexpected termination
  • 478416 Peach timeout during desktopui_FlashSanityCheck

CrOS gardener: 
Sheriffs: drinkcat, snanda, tbroch

canary failures:
  • 529905: bobcat: setup_board fails
  • 523313: nyan_blaze: autoupdate failed. (#2084 / #2085)
    • Happened twice in a row now. Might be worth investigating further.....
  • 529044: platform_Powerwash: leon does not reboot?
    • Assigned to deymo
  • 529216: x86-mario Paygen: Failed with Killing tasks: [<_BackgroundTask
    • 529113: stumpy-moblab (moblab_SmokeSuite)
      • Long standing issue
    • 529674: canary master fails in BranchUtil
      • Assigned to vitalybuka
    • 529428veyron_pinky: 'str' object has no attribute 'write'
      • Assigned to pprabhu
    • 529565: The Paygen [stout] stage failed: <class 'chromite.lib.paygen.gslock.LockNotAcquired'>
    • 529466: x86-mario Unhandled AutoservRunError: command execution error
    • 529324: butterfly/sumo provision failure
      • Race between UploadTestArtifacts and HWTest phases
    • 529612: akitu_mobbuild: cloud_CloudInit fails in VMTest (twice)
    • 529480: Several PFQs failing on security_SandboxLinuxUnittests
      • Fix landed in chromium. Ensure that the subsequent runs pass.
      • This should be fixed from 47.0.2506.0, but PFQ is not picking it up (I tried to abort the current build to force it... that did not work...). Looks like we need to wait for the release tag 47.0.2506.0 to appear on (should be Wed 8pm PST if I understand correctly).

    CrOS gardener: 
    Sheriffs: drinkcat, aaboagye, benzh
    • 529480: Several PFQs failing on security_SandboxLinuxUnittests
      • Fix landed in chromium. Ensure that the subsequent runs pass.
    • 525128: CustomLauncherPageBrowserTest.EventsActivateSwitchToCustomPage browsertest failing on Builder: Linux ChromeOS Buildspec Tests
      • This started failing on the 6th. Need to find a fix ASAP.
    • 529388: ninja failed in build_packages: sys-kernel/chromeos-kernel-3_10
      • The subsequent build seemed to work fine.
    • 528567: daisy pool critical state (board does not recover from update to 7426.0.0, 7427.0.0 or 7428.0.0; 7429.0.0 seem ok)
      • 529443: daisy provision_AutoUpdate failures
    • 528748: x86-mario pool critical state => Similar to above
      • There were some shill issues and DUTs going down. See b/23896777.
    • 524814: daisy/x86-mario Paygen => Long standing issue. Possibly causing 2 issues above?
      • 518591: zako/clapper: Could not find *_full_* in Google Storage (possibly same root cause)
      • 529216: veyron_mighty/nyan_blaze: Failed with Killing tasks: [<_BackgroundTask (possibly same root cause)
    • 529113: stumpy-moblab (moblab_SmokeSuite) => Long standing issue
    • 487955: daisy full failure in ChromeSDK => Old issue resurfacing (missing dependency?), check if happens again on next run
    • 529160daisy_spring failed to return from powerwash (log) => Check on next run if it happens again
    CrOS gardener: 
    Sheriffs: jcliang
    • 516795: Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
      • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
    • 524814: Infrastructure Issues (code 3 and code 1) 
    CrOS gardener: 
    Sheriffs: jcliang
    • 516795: Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
      • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
    • 524814: Infrastructure Issues (code 3 and code 1) 
    • 528176: vmtest fails logging_UserCrash on x86-generic incremental, x86-generic ASAN, and amd64-generic ASAN
    CrOS gardener: 
    Sheriffs: cywang
    • 516795(merged 526629): Many group canaries has the issue of failures "too close to timing out, or exceeds the timeout"
      • root cause: Test Infrastructure issues (takes a long time in HWTest(bvt))
    • 509779 Infrastructure Issues (code 3 and code 1) 
    • 526641(fixed): after vapier clobbered ccache, sys-apps/util-linux-2.25.1-r1 built successfully
    • 527909: veyron_jaq canaries failing to sign/paygen due to broken bsp
    • 528017: veryon_jerry canaries failing to sign/paygen due to broken bsp
    CrOS gardener: 
    Sheriffs: cywang
    • 526716 WebRTC is failing to build on mipsel. Preventing up rev of Chrome on CrOS
    • 526629: HWTest [clapper] [bvt-inline] test timeout failure?
      • test seems finished after the test was stopped by 'timeout'?
    • 526641: pineview group: failed to build sys-apps/util-linux-2.25.1-r1
    CrOS gardener: 
    Sheriffs: dhendrix
    • 524814: Canaries are falling over on autoupdate_EndToEndTest.paygen_au_canary_test_full
    • Provision failures that seemed to fix themselves.
      CrOS gardener: tbrazic
      Sheriffs: ejcaruso, bfreed, hychao
      • 517876: DUTs lost RPC connections
      • 523189: login_OwnershipTaken failures on multiple boards

      CrOS gardener:
      Sheriffs: ejcaruso, bfreed, itspeter
      • 522851: [Pri-0] Google cloud storage exception causing archive steps fail across major builtbot
        • Closing tree because this cover all the errors underground.
        • reverts which changed how we build and install rsa_id files.
        • Note: We cannot login to bots to confirm the missing file, but the log entry "CommandException: No URLs matched: /b/cbuild/external_master/buildbot_archive/daisy-incremental/R46-7381.0.0-b24819/id_rsa" gives the best clue.
      • 522785: buildpackages [x86-alex] [afdo_use] is flaky in pineview group canary
      • 518591: samus-release: provision failure, infra flaky, succeed in next build.
      • 503526: ivybridge-freon-release-group: DUTs pool is too small
      • 496036sandybridge-release-group: DUTs pool is too small
      • 523170: gizmo canary fails during BuildPackages
      • 523173: nyan group canary timed out during paygen
      • 523174: auron group canary has issues with rpm unit tests
      • 523139: sandybridge-freon group canary, x86-alex_freon paladin not found on in master and won't build

      CrOS gardener: girard
      Sheriffs: dlaurie, wiley, itspeter
      • 522533: lab problems creating various failures
      • 522540: amd64-generic and x86-generic chromium PFQ failures in security_OpenFDs test
      • 522528: HWTest run_suite failures
      • 522410: ap-daemons fails to build on storm, blocking many pre-cq runs
      CrOS gardener: girard
      Sheriffs: dlaurie, wiley, owenlin
      • Spontaneous network failure! Tests won't succeed without network.
      • 522130: LKGM timeouts on Chrome PFQ (fixed)
      • 522139: Paygen timeouts
      • 522141: sandybridge-freon-release-group builder needs to be removed
      • 522147: mario-incremental failing (fixed)
      CrOS gardener: girard
      Sheriffs: davidriley, waihong, owenlin
      • 521642: daisy-skate build failed on database error
      Sheriffs: davidriley, waihong, kcwu
      • 520931: provision Failure on samus-release, looks like hardware flaky.
      • 521046: VMTest kernel_CryptoAPI failed on lakitu canary
      • 521018Several canary groups timed out on the step "steps" (no attribute 'PrintBuildbotStepFailure')

      Sheriffs: jrbarnette, armansito
      • Multiple canary failures in the AM, especially 520311.
        • Pinned Chrome; waiting for the overnight canaries to prove whether that fixed it.

      CrOS gardener: michaelpg
      • 516978: No space left on device -> stateful partition sizeincreased
      • 518015: Bots haven't signed the new CLA; LKGM candidates not uploading -> CLA enforcement rolled back
      Sheriffs: cychiang, deymo, adlr
      • 517308: security_test_image faild with wrong fs type to mount recovery_image.bin. --> The image seems good and running security_test_image locally can pass.
      • 517388: mipsel-o32-generic full failed at ChromeSDK due to --hash-style defaulting to gnu
      • 517348: [paygen_au_dev] autoupdate_EndToEndTest.paygen_au_dev_test_full Failure on peach_pi-release/R46-7335.0.0
      • 517351: [sanity] provision Failure on lumpy-release/R46-7335.0.0 stage_artifacts timed out
      • 517460build_package failed at chromeos-chrome
      Chrome Gardener: stevenjb
      • 517238: RESOLVED: ExtensionTestMessageListener::WaitUntilSatisfied() causing flake across a large number of tests
        This turned out to actually be 515914 - browser_tests step fails even though all tests pass
      • 516978: P0 STILL IN PROGRESS: piglit file collision cause build_image failure. --> No space left on device
        This is causing PFQ failures
      • 517593: Frequent browser_tests time out with (TIMED OUT) in log
        This isn't causing any detected failures because the tests get retried, but it does slow down the tests a little and generates confusion when other issues (e.g. 
        515914) show up.

      Sheriffs: cychiang, dianders, denniskempin
      • 516978: piglit file collision cause build_image failure. --> No space left on device
      • 493752: Lab DHCP failures lead to "Host did not return from reboot". It causes PFQ build failure.
      • 517027: jecht and veyron_pinky not available in the lab.
      • 516795: veyron group canary is too close to its timeout. ---> seeing other boards fail of the same reason.
      • 514700: Samus failures due to stateful partition of Samus DUTs being bad.  ext4_lookup errors, "cannot remove" errors, etc.
      • 515880: No more samus devices in lab that are good (probably because of above bug).
      Sheriffs: djkurtz, dianders, denniskempin
      • 488291: Looks like vm_disk: failed_SimpleTestVerify is flaky and  throttled the tree on amd64-generic ASAN. 
      • chromeos-hwid broke in paladin several times.  Probably <>.
      • 516750: Samus failures due to stateful partition of Samus DUTs being bad.  ext4_lookup errors, "cannot remove" errors, etc.
      • FYI: binutils roll happening
      • 515528: AU tests failing.  I don't think they always ping this bug, though...
      • 516793: cbuildbot timeouts are hard for sheriffs to decipher.
      • 516795: veyron group canary is too close to its timeout.
      • 530203 Samus with on-going HWTest Aborts.

      Sheriffs: djkurtz, moch, vbendeb
      • 516283: chromite: Unittest failure on veyron_pinky in mobmonitor/checkfile/manager_unittest: URLError: <urlopen error [Errno 111] Connection refused>
      • 516286: chromite: Unittest failure on "auron group canary" in sync_stages_unittest timeout?
      • 516295: zako BuildPackages fails in media-libs/fontconfig
      • 515528[paygen_au_canary] autoupdate_EndToEndTest.paygen_au_canary_test_delta Failure on falco_li-release/R46-7315.0.0
        • The autoupdate_EndToEndTest.* tests seem to be failing across many boards over the past ~3 days:
      • 516371Signer test failing due to unexpected kernel parameter (CL reverted)
      • 516430x86-alex paladin: bvt-cq timed out
      • 516457HWTestStage fails with
      • 488291amd64-generic ASAN failing on desktopui_ScreenLocker
      Sheriffs: vbendeb, moch, pprabhu
      • 5155Sheriffs: 60CrOS Tree Closure: ChromeOS killing DUTs in the lab
      • 515905SyncChrome fails on all Chrome PFQ (non-issue - caused due Chrome pinning because of the above issue, PFQ expected to fail)
      • 515937Temporary workarounds in lab to get DUTs off of a bad build
          2015-07-29 and 2015-07-30
          CrOS Gardener: alemate (29), afakhry (30-31)
          Sheriffs: pprabhu, grundler, zuethen, 
            • 501178: 13 groups timeout... HWTest of many boards are not running. Closed the tree.
            • 515479: kvm is missing on a lot of precq bots breaking vmtest (Fixed by davidjames)
            • 515201: Chrome crashes (AddInputDevice) affecting other services(Fixed in Chrome; see also 515154 43375)
            • 514700: Samus: file system corrupted after Kingston FW update (reverted Kingston FW update)
            • 434755: daisydog constantly restarting (Fix in CQ - x86_64 v3.8 kernel missing /dev/watchdog since last fall)
            • 515576: shill crashes took a bunch of DUTs "out of service": alex/zgb/peppy/lumpy/...  (chumped Fix in shill)
            • HW TestLab Network failure: outbound traffic was losing 60% or more packets (subsided; still monitoring)
            • 515905: SyncChrome fails on all Chrome PFQ (Chrome pinned to old version?)
            • 515302SitePerProcessBrowserTest.DiscoverNamedFrameFromAncestorOfOpener is failing on the official cros-trunk.
            • 515567: Compile failures on cros-trunk due to #include the generated header "ui/gfx/vector_icons2.h".
            • 516052: More Flaky tests in AutofillInteractiveTest on the official cros-trunk.
            2015-07-27 and 2015-07-28
            CrOS Gardener: bruthig, alemate
            Sheriffs: avakulenko, abrestic, vapier/henrysu
            • 514257: Lost a bunch of DUTs due to AFE going down last night
            • 514364: coreboot branch update caused some builders to go red.
            • 512996 BrowserEncodingTest.TestEncodingAutoDetect timing out on cros_trunk
            • 513593: [bvt-inline] security_SandboxStatus Failure on lumpy-chrome-pfq/R46-7296.0.0-rc1
            • 514401 Multiple build's are failing syncchrome with error: "reference is not a tree: 8a0429e414914781450ca007f20b0e511b3acff7"
            • 514499 provision_AutoUpdate failures on Rambi
            • 460860 more login_Cryptohome failures
            • 514504 desktopui_ScreenLocker failure on Auron (flake?)
            CrOS Gardener: bruthig
            • 513593: [bvt-inline] security_SandboxStatus Failure on lumpy-chrome-pfq/R46-7296.0.0-rc1
            CrOS Gardener: jonross
            Sheriffs: quiche, shchen
            • 512417 New FecSendPolicy test is crashing on CrOS Trunk
            • 512427 Chrome PFQ failing on autotest-tests-ownershipapi
            • 512435 Canary board timeouts, missing boards for HWTest
            • 465862 Flake in desktopui_ScreenLocker, with ASAN
            • 509274 Canary timeout on candy
            • 512577 CrOS Trunk failing on ClickModifierTest
            • 513618 mipsel-o32-generic-full failed building Chrome

            CrOS Gardener: jonross
            Sheriffs: bleung, hungte, shawnn
            • 512010 Chrome OS Canary failure in BranchUtilTest failures: no such option --nobuildbottags
            • 512024 Master release failure
            • 511680 Samus HWTest failure
            • 491361 Strago VMTest failure
            • 508637 Jecht-family failure
            • 511542 winky DUT shortage
            • 512174 CrOS Commit /Queue HWTest failures: /var/log/storage_info.txt does not exist

            Sheriffs: bleung, shawnn
            • Known-issues causing multiple canary failures:
              • 484726: autoupdate_Rollback failing, possibly due to DNS issues in lab
              • 508637: rikku-family stuck at login screen
              • 510909: Paygen kernel hash issue. Should be resolved.
              • 505744: AutoservSshTimeout
            • 511317: login_OwnershipTaken timing out (silently?) 
            • 511502: libstrongswan missing symbols
            Sheriff: dbasehore, alecaberg
            • 510759: Paygen lock not acquired
            • 481092: builder must call SetVersionInfo first
            • 510909: paygen failure, kernel hash doesn't match
            • 509837: amd64 ASAN flake

            Sheriff: zqiu, marcheu, wuchengli
            • 510074 amd64-generic-llvm builder unittest failures
            • 509779 Flaky HWTest failures
            Sheriff: furquan, charliemooney
            • 465862 Flaky screen lock test
            • 508637 Rikku: Login screen problem in HWTest
            • 491290 Flaky SSH Failure

            Sheriff: puthik, rspangler, seanpaul
            • 507279 Lumpy/Falco pfq hwtest failing on timeout 
            • 501966 pool: bvt, board: lulu in a critical state
            • 507372 Drone refresh/execute took over 50s
            • 470701 Flaky BVT security_Firewall failure, "Mismatched firewall rules"

            Sheriff: posciak
            • 505918 CheckFileModificationTest timeouts
            • 506030 Payload generation failures
            • 506037 autotest-chrome build failures on missing dependencies

            Sheriff: cernekee, reinauer, kpschoedel
            • 505108: wolf-paladin and wolf-canary are failing, lab is closed.  Somebody will fix this Monday.
            • 505051: "Mismatched firewall rules" test failure on x86-generic build
            • 504947: HWTest failures on ivybridge, daisy
            • 504861: ASAN buildbot failures
            • 504860: HWTest did not complete due to infrastructure issues
            Sheriff: cernekee, reinauer, wiley
            • 504602: builders cannot get veyron_pinky chrome prebuilts
            • 504476: pre-cq builders are not running
            • 504400: Crash in SpokenFeedbackEventRewriterDelegate
            • 472895|AFDO generate| should only run if Chrome has changed
            Sheriff: dtor, wiley, gedis
            • 465862amd64-generic ASAN failure (desktopui_ScreenLocker fails - hitting regularly)
            • 430836: autoupdate_Rollback failure
            • 491598: platform_Powerwash flake
            • 412795: Refresh Packages is down
            • 488580: image_to_vm failing
            • 472895: Canaries failed while syncing Chrome
            Sheriff: bowgotsai
            • 488291flaky on login_LoginSuccess test
            • 465862amd64-generic ASAN failure
            • 503188: HWTest failure
            • 488580parrot canary: image_to_vm failing

            Sheriff: bowgotsai
            • 488291flaky on login_LoginSuccess test
            • 465862desktopui_ScreenLocker keeps failing on x86-generic ASAN VMTest
            • 502897: AU test failed on "No module named netifaces"
            • 502909Multiple HWTest failures
            • 502910CommitQueueSync failure
            • 503001: cbuildbot command failed on multiple PFQ builders

            Sheriff: josephsih
            • 500640Multiple HWTest failures
            • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
            • 481092: ManifestVersionedSync: RuntimeError: builder must call SetVersionInfo first
            • 483661: x86-generic full: vmtest failed in SimpleTestUpdateAndVerify
            • 501178CanaryCompletion: 20 groups timed out
            • 444876Clear and Clone chromite: remote: User Is Over Quota

            Sheriff: filbranden
            • 500640Multiple HWTest failures
              • Hardware is borked and we had a deputy outage, so we had to work around it by disabling the hw_tests that were failing.
              • CL 277672 disabling the tests.
              • ChromeOS Infra team to revert that CL once the hardware is working again.
            • 500394build_package fails on several PFQ builtbot since 6/12
            • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
              • Still open, if it keeps the tree closed we need to prioritize a revert (even possible?) or push a rushed fix.
            Sheriff: itspeter
            • 500394build_package fails on several PFQ builtbot since 6/12
            • 500423: Payload integrity check failed: install_operations[490](MOVE): MOVE operation cannot have extent with start block 0
            Sheriff: itspeter
            • amd64-generic ASAN VMTest failure
              • 465862Build haven't been succeed since 2015-06-10, desktopui_ScreenLocker keeps failing
            • x86-generic ASAN VMTest failure
              • 488291Seems to be flaky on login_LoginSuccess test
              Sheriff: tyanh
              Gardener: girard
              • Infra failures on Canary bots
                • 491290: autoupdate_Rollback Failure on rambi-release/R45-7142.0.0
                • 497035: autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on link-release/R45-7142.0.0
                • 497059: HWTest suite prep aborted on Stumpy_moblab Canary
              • 497092: Chrome PFQ failing at BuildPackages across all builders - reverted patch - next build should be okay
                Sheriff: tyanh
                Gardener: girard
                  • Infra failures on Canary bots
                    • 496552: [au] autoupdate_Rollback; Unhandled AutoservRunError: command execution error
                    • 496526: [sanity] provision; Unhandled HTTPError: HTTP Error 500: Internal Server Error
                    • 476324: HWTest provision failure=
                    • 460925: HWTest/sanity provision; Unhandled TimeoutException: Call is timed out
                    • Host did not return from reboot
                  • 496523 mipsel fails continuously on PFQ

                  Gardener: jonross
                  • 460925 Chrome PFQ seeing Infra failures in HWTest, assigned to Infra
                    • Affecting: daisy_skate, falco, lumpy, peach_pit, 
                  • 465162 469495 Infra failures blocking the PFQ
                  • A series of Infra failures for Chrome OS Canary bots were auto-filled. Not enough bots in the pool
                  • CL 274236 removed two bots from the Waterfall x86-generic-tot-chrome-pfq-informational, amd64-generic-tot-chrome-pfq-informational)
                  • 496273 mipsel-032-generic having gyp error preventing a build.
                  • 496293 Chrome OS PFQ is trying 45.0.2421.0 which does not contain the fix to 494912 whioh would unblock the PFQ
                  • 496325 Flaky OAuth test on Linux Chromium OS
                  Sheriff: robertshield, aaboagye, ssl
                  Gardener: jonross
                  • Chrome PFQ failures still holding up upreving.
                    • 494041494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
                      • Test was landed in this change assigned to developer who landed the tests
                      • Seems like an actual regression in video playback for ARM.
                      • Regression is from Chrome side between 45.0.2416.0 and 45.0.2417.0. Reverting the video changes in this diff does not seem to fix the issue. Trying to revert the WebRTC roll, but its failing locally. I've reached out to the WebRTC sheriff
                      • Suspects 80f289fe303323361d07c5b58b23f8499903a154c794eda78e9ba3c46b550b433e9fe5a248d40104 as the bug is not present with hardware acceleration off. Apparently my local build has an issue, as I still have the failure with hardware acceleration off.
                    • 413961: falco failed on stateful_update. It looks like the download got interrupted.
                  • Canary Failures
                    • I believe these were network related... some are reporting high flakiness
                    • 476368495463428345 have been filed against the Canary failures.
                    • 493219: nyan group failed with Connection reset by peer.

                  2015-05-30/31, 2015-06-01
                  Sheriff: ssl, aaboagye
                  • 482284: ivybridge-freon-release-group: The BuildImage [stout_freon] [afdo_use] failed - cannot open ‘/dev/loop0p1’ for reading: Permission denied
                  • p/40797: oak fails to cross-compile img-ddk properly
                  • 477928: (quawks, x86-zgb) autoupdate-Rollback - ssh: Could not resolve hostname chromeos2-row2-rack7-host6: Name or service not known (network went away briefly?? DNS issue?)
                  • 493533: asan bots failed quipper unittests. x86-generic ASAN has been broken since 5/28.
                    • Mainly due to the quipper unittest failing, but also due to login_LoginSucess failure. See 488291.
                    • Waiting on this CL.
                  • Chrome PFQ failures
                    • 494041, 494912: daisy_skate && lumpy PFQ failing on video_VideoSanity.
                    • 494909: lumpy PFQ failing on desktopui_FlashSanityCheck.
                  • 495281: One off wolf-tot-paladin failed during HWTest [sanity] - provision: ABORT: reboot command failed
                  • 493718: Chrome PFQ Failing to uprev Chrome commit - on tricky, which is experimental is a mistake. Disregard these emails.
                  • 493219: Some canaries failed due to FAILED RPC CALL. (beltino-a/b group, sandybridge-freon, ivybridge-freon) Manifested as timed out and Connection reset by peer.
                  • CL:274726: Fix lakitu build_image error when modifying kernel command line.

                  OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)