Sheriff Log: Chromium OS (go/croslog) (ARCHIVED)


Please update go/cros-sheriff-playbook when you find a build/infra failure and can map it to what action the sheriff should take for it.

10/15-10/19
Sheriffs: seobrien, amstan, josephsih
Ongoing issues:
  • crbug.com/895203: caroline-arcnext-paladin: Not enough DUTs (fixed)
  • crbug.com/895215A few platforms: HWTest failed: ui.ChromeLogin (fixed)
  • crbug.com/895221guado: HWTest failed: video_WebRtcMainFeedSwitching (kerl@ had a fix. Eventually, should remove bluestreak-pre-cq from HWTest)
  • crbug.com/895241amd64-generic-paladin: TastVMTest failed: platform.CheckProcesses (assigned to semenzato@)
  • crbug.com/895247: Many (66) builders: failed with _ShutDownException (Won't fix.)
  • crbug.com/894820: amd64-generic-paladin UploadTestArtifacts Failures (still seeing this issue frequently)
  • crbug.com/896158: hana-paladin: Not enough DUTs (assigned to akeshet@)

10/08-10/12
Sheriffs: saklein, yueherngl, tbroch
IRC: 10/08
Ongoing issues:


8/20-8/24
Sheriffs: tfiga, bleung, akhouderchah
Ongoing issues:
  • crbug.com/875710 - bob-paladin: login_RetrieveActiveSessions: fails with "Can't create session"
  • crbug.com/873687 - Bob-paladin: logging_CrashSender fails in policy signature verification
  • crbug.com/876186 - cyan-paladin:6819 failed: cheets_Midis: midis didn't restart.
  • (Resolved) crbug.com/876188 - edgar-paladin:3720 failed: Failed to start ARC: failed waiting for ro.arc.boot_completed=1
  • crbug.com/876196 - wizpig-paladin:3846 failed: login_LoginSuccess: Timed out waiting for condition: Session stopped.
  • crbug.com/876197 - wizpig-paladin:3846 failed: login_GuestAndActualSession: Unhandled DBusException: org.freedesktop.DBus.Error.NoReply
  • crbug.com/876236 - master-paladin:19462 failed: Cannot find overlay: [...]/overlay-grunt-ndktranslation-private
  • crbug.com/876238 - tatl-paladin:4295 failed: InvalidLayout: Unknown items in layout recovery
  • (Resolved) crbug.com/876630 - shard heartbeat queries timing out all over the fleet
  • crbug.com/876634 - nyan/peach/veyron/elm/hana: multiple graphics_/login_ HWTest failures due to missing python modules
  • crbug.com/876988 - winky-paladin:5905 failed: security_StatefulPermissions: Unexpected files/perms in stateful
  • (Resolved) b/113088029 - Boot time regression of ARC++ in VMTests?
  • crbug.com/877342 - master-paladin: SQL error in CommitQueueHandleChanges if all builds are cancelled?
  • (Resolved) crbug.com/877372 - logs.chromium.org returns 503 Over Quota
  • crbug.com/877390 - edgar: All DUTs in pool:cq are "Repair Failed"
  • crbug.com/877394 - wolf-paladin: Provision failed on all DUTs
8/13-8/19
Sheriffs: moragues,jwerner
Ongoing issues:
  • 871513: Multiple release/paladin failing with " [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s"
  • crbug.com/874705 - Unnecessary kevin-paladin & kevin-arcnext-paladin failures due to the share of DUTs
  • EXPERIMENTAL=coral-paladin crbug.com/866543 process leak caused : NotEnoughDutsError: Not enough DUTs for board: coral, pool: cq; required: 4, found: 3
  • 873822: Kernel warning spam on RK3399 devices causes disks to fill up, leading to all sorts of random failures. Affects mostly Bob since Kevin has larger disks. 
    • EXPERIMENTAL=bob-paladin crbug.com/873687 Bob-paladin: logging_CrashSender fails in policy signature verification
8/6-8/12
Sheriffs: jkwang, kirtika, stimim
Ongoing issues:
  • 871513: Multiple release/paladin failing with " [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s"

    7-30-8/3
    Sheriffs: teravest, evgreen, 

    Ongoing issues:
    • 868331: dragonegg paladin: Detected file collision - tpm_version
    • 867319: Not enough DUTs for board: daisy_spring, pool: bvt; required: 4, found: 3 (also b/111838841)
    • veyron_rialto: Known issue, not enough DUTs
    • falco_li: Known failures, not enough DUTs
    • 870787: samus-release: key_verify failed for server_host_key
    • 870511: squawks-release failures: graphics_idle: Short blanking not added
    • Untriaged: Some (many?) release builders are failing with [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s
    • b:112082609: kefka-release: Device not coming back after reboot
    • 869952: security_SandboxLinuxUnittests: test does not exist
    • 870005: nami-release failed cheets_CameraOrientation
    7/23-7/27
    Sheriffs: sjg, shurst, wuchengli
    Ongoing issues:
    • 867319: Not enough DUTs for board: daisy_spring, pool: bvt; required: 4, found: 3 (also b/111838841)
    • 867320: newbie-release: Unreasonably low CPU frequency in cheets_SettingsBridge (now replaced by crbug.com/867649 to get rid of newbie!)
    • 867341: kefka-release: flaky "Host did not return from reboot"
    • 867357: master-paladin: PublishUprevChanges failed
    • 868331: dragonegg paladin: Detected file collision - tpm_version
    Resolved issues:
    • 865064 - Lotsa dut pools be unhappy, causing problems for veyron_rialto, banjo, etc.
    • 867317: Not enough DUTs for board: falco_li, pool: bvt; required: 4, found: 2
    • 867318: veyron_jaq-nyc-android-pfq failing in UprevAndroid
    • 867360: daisy-incremental: Resolving storage.googleapis.com... failed
    • 867326: Not enough DUTs for board: veyron_rialto, pool: bvt; required: 1, found: 0

    4/23-4/27
    Sheriffs: jrio, gwendal, yhong
    Ongoing issues:
    • b/78599178Git on Borg (GoB) issue, git fetch fails.
    • 837388cros-full servers overloaded, leading into HW Test suite timeout.
    4/16-4/20
    Sheriffs:
    nshai, lepton, hychao
    Ongoing issues:
    • 831624eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
    • 832880: rmi4update kills chrome on nautilus
    • 831732: scarlet-release: Not enough DUTS available - need to remove temp. experimental from chromiumos-status when fixed!
    • 827347: security_SandboxedServices test failure on stout due to root run dbus-monitor in rf-led-handler.conf (EOL device)
    • 795128: evemu-device did not die resulting in security_SandboxedServices failing
    • 796254: Archive stage fails in build_image
    • 833478some autotest_lib.site_utils.lxc.* unittests are flaky.
    • 547055: tar: file changed as we read it.
    Resolved issues:
    • 833886: Data base error cause most builder error.
    • 833504: reef release broken.
    • 833456: Chrome crashes when running in mash on device.
    • 833499: nami release broken.
    • 83032167: Caroline and Terra builds are RED; Chrome crashes at boot
    • 829780: fizz R66 hwtest failures (waiting for devices in lab to get new base image flashed) - need to remove temp. experimental from chromiumos-status when fixed
    • 831391tatl-paladin:3250 failed: cannot find wayland-scanner
    • 832280: android-container-nyc failing BuildPackages on an unrelated CL
    4/9-4/14
    Sheriffs:
    pberny, dianwa, sheckylin
    Ongoing issues:
    • 831624eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
    • 83032167: Caroline and Terra builds are RED; Chrome crashes at boot
    • 829780: fizz R66 hwtest failures (waiting for devices in lab to get new base image flashed) - need to remove temp. experimental from chromiumos-status when fixed!
    • 831391tatl-paladin:3250 failed: cannot find wayland-scanner
    • 832880: rmi4update kills chrome on nautilus
    • 831732: scarlet-release: Not enough DUTS available - need to remove temp. experimental from chromiumos-status when fixed!
    • 831624: eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
    • 832280: android-container-nyc failing BuildPackages on an unrelated CL
    • 827347: security_SandboxedServices test failure on stout due to root run dbus-monitor in rf-led-handler.conf (EOL device)
    Resolved issues:
    • 830865: PFQ builds failing - SyncChrome failure: ParseDepsFile:Found recursedeps
    4/2-4/6
    sheriff: smbarber, zhengpan, mnissler
    Ongoing issues:
    • 828371: Not enough DUTs for board: eve
    • 828054: sporadic build_image failures
    • 829055: whirlwind build flakiness
    • 829289: CanaryCompletion failing on multiple builders
    • 824808intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'
    • 829780: fizz R66 hwtest failures

    3/19-3/23
    sheriff: eugenegonzalez, shawnn, akahuang
    Ongoing issues:


      3/5-3/12
      sheriffs: mgild, nvaccaro, itspeter
      Ongoing issues:
      • 813916: "Not enough DUTs for board: expresso"
      • 815308: coral-release fails HWTests; probably compound flakiness from having to pass on 13 different boards in the lab. Unsure what we can do about this.
      • Investigate on master-paladin failure. It will fail with CL:*581566, ask the author to take a look. 
        • Update: owner submit again after other dependent CLs merged. It is merged as well now.
      • 819576: moblab-generic-vm-paladin seems flaky and failing mater-paladin
      • 819695: Merge 819576 to this one as this is actively investigating.
        • moblab-paladin flake: ERROR: Unhandled UpstartServiceNotRunning: Upstart service moblab-gsoffloader-init not in running state. 

      2/26-3/2
      sheriffs: ejcaruso, matthewmwang, wnhuang
      Ongoing issues:
      • 789058: CreateTarball race in DebugSymbols step of several release builders
      • 813916: "Not enough DUTs for board: expresso"
      • 814343: daisy_skate-release: NotEnoughDutsError
      • 814347: falco_li-release: NotEnoughDutsError
      • 814352: nautilus-release fails on cheets test ( adb is not ready in 60 seconds.)
      • 814500: scarlet-release: NotEnoughDutsError
      • 815308: coral-release fails HWTests; probably compound flakiness from having to pass on 13 different boards in the lab. Unsure what we can do about this.
      • 817074: chromeos2-row8-rack2-host5 is in a bad state, causing setzer-release failures
      • 817437: cros-beefy280-c2 is offline
      • 817948: soraka usb/ethernet instability, likely product issue
      Resolved issues:
      • 814345: falco_release: NotEnoughDutsError
      • 815250: newbie-release fails Signing stage and no email notification is generated
      • 816563: ultima-release: NotEnoughDutsError
      • 816584: enguarde-release: NotEnoughDutsError
      • 816983: guado-release failed from one DUT having issues updating
      • 816986: reef-release won't get past CleanUp stage after build crash
      • 817022: veyron_tiger-release is not running
      • 817063: poppy DUTs in verify-repair loop
      • 817126: chromeos2-row3-rack1-host13 is in a bad state, causing soraka-release failures
      • 817478: 2-3-1-9 is down, causing soraka-release failures
      • 817925: fizz EC fails to sign

      2/19-2/23
      sheriffs: norvez, waihong, rongchang
      Ongoing issues:
      • 814340: platform_addPrinter is flaky (low frequency), sometimes hits the paladins
      • 812425quipper build seems to be failing unittests occasionally
      • 814500: scarlet-release: NotEnoughDutsError
      • 814343: daisy_skate-release: NotEnoughDutsError
      • 814345: falco_release: NotEnoughDutsError
      • 814347: falco_li-release: NotEnoughDutsError
      • 813916: "Not enough DUTs for board: expresso"
      • 814352: nautilus-release fails on cheets test ( adb is not ready in 60 seconds.)
      • 815250: newbie-release fails Signing stage
      • 815308: coral-release fails HWTests - DUTs look really unstable
      • 798618: Intermittent paladin failures with cheets_StartAndroid.stress
      • 813791: Changes to chromeos-base/trunks cause build failures in CQ
      Resolved issues:
      • 814514: veyron_rialto-release: No module named telemetry.core

      2/12-2/16
      sheriffs: snanda, mqg, littlecvr
      Ongoing issues:
      • 811210: HWTest failed due to infra issue (code 3)
        • 811149: widespready DUT pool shortfalls | shards unable to resolve DUTs | dhclient_conf is not set to "yes" on new shards
      • 811217eve-arcnext-mst-android-pfq:78 failed (PackageBuildFailure: Packages failed in ./build_packages: x11-libs/arc-libdrm)        
      • b/72697187: quota increase request for ChromeOS Infrastructure
      • 811402: Master scheduler crashlooping because of malformed HQE
      • 811878cheets_KeyboardTest fails frequently on multiple boards: retry_count: 2, FAIL: Unhandled AssertionError
        • 569819: temporarily remove cheets_KeyboardTest from CQ <- already reverted
      • 812425quipper build seems to be failing unittests occasionally
      • 812949: DUT not rebooted in provisioning
      • 798618: Intermittent paladin failures with cheets_StartAndroid.stress
      • 812581: PackageBuildFailure: Packages failed in ./build_packages: chromeos-base/chromeos-chrome
      • 812848: DUTs get stuck in un-abortable state
      • 811697: Cyan HQEs queuing up catastrophically in shard (M66: Caroline, Cyan build is RED for 2 days)

      2/5-2/9
      sheriffs: craigb, adlr, abhishekbh, shenghao
      Ongoing issues:
              808923: Chrome PFQ failed due to not able to download files from GS
              808945: HWTest failed due to infra issue (code 3)
              809570: edgar-paladin can't mount chroot
              809670CQ blocker: nyan_kitty-paladin fails due to video test errors
              810247wolf-paladin fails: Start browser timeout
              810255: cheets_ContainerMount failed and is blocking CQ
              810667coral-paladin failed on pack_firmware_unittest.py

      1/29-2/2
      sheriffs: vpalatin, ravisadineni, abhishekbh
      Ongoing issues:
              
      808434: Canaries are failing at the build_packages stage due to a python error.
              808563: bvt-arc suite in Canaries builds failing due as the server side tests do not start after scheduling.

      1/22-1/28
      Sheriffs: bmgordon, shapiroc, marcochen

      Ongoing issues:
      • 784914: provision failurs: DUT cannot reboot at pre-setup of rootfs update
      • 806287: chromeos4-row11-rack10-host13 is failing to provision
      • 806013: libbrillo has new ASAN error
      • 796275bvt-arc times out across boards
      • 804977: guado_moblab-paladin is failing for HW tests
      • 782832: not enough daisy_skate devices to keep bvt pool alive
      • b/72397774: intermittent failures to connect to git/gerrit
        • 805928: Release builders failing during ManifestVersionSync stage
          • This looks like potentially a transient git issue
      • 782034: autotest artifacts persist between CQ runs
      Resolved issues:
      • libc++ related changes
        • 805691: qhull fails to build with libc++
        • 805619: buildpackages failing on numerous packages in bare precq run
        • 805722: Clobber incremental builders
        • 805657: arc-camera3-hal-intel-ipu3 fails to biuld with libc++
      • TKO/mysql issues
        • 806019: tko query pileup | tko restart takes over an hour
        • 805337: TKO database reached maximum size.
        • 804127: shard outage for board:leon, board:nyan_big
        • 805724: job_reporter died, causing passing test to appear aborted
        • 804425: shard outage for board:orco
        • 806011: chromeos-server133 afe serving lots of 5XXs
      • 806106: lumpy-incremental-paladin failure
      • 806107: Not enough lumpy DUTs
      • 806196: Uprev stage fails on cros_mark_as_stable
      • 805517: Betty android pfq tests are failing due to VM AU issue
      • 805710: gale-paladin fails in ap-daemons unit tests
      • 804513: eve-paladin failed to rebuild previously removed ebuild
      • 804372: Missing alerts in Sheriff-o-Matic

      1/8-1/12
      Sheriffs: ahassani, dianders, hiroh

      NOTE: Trying idea of just keeping the week's log in a Google doc.

      Ongoing Issues:
      • 782832: not enough daisy_skate devices to keep bvt pool alive
      • 795902: fizz-release failing since Dec 8
      • 800831: provisioning issue across the board
      • 800943: ls-remote-gerrit rate limit exceeded
      • 800949: canaries had a problem pushing to gerrit
      Resolved Issues:
      • 800426race between the Android PFQ and the CQ
      • 800132: infrastructure issue caused provisioning failures
      • 800886: moblab-paladin autotest failure

      1/1-1/5
      Sheriffs: frankhu, sarthakkukreti, johnylin

      Ongoing Issues:
      • 782832: not enough daisy_skate devices to keep bvt pool alive
      • 795902: fizz-release failing since Dec 8
      • 788584: kefka/coral-release/paladin: Linksys USB3GIGV1 Ethernet adapter fails to enumerate (r8152, usb X-Y: device not accepting address Z, error -62)
      • 797849: paygen tests from previous canary delayed bvt-arc from canary, causing timeout
      • 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed
      • 799669banon-release: build fails security_SandboxedServices
      • 799604: Lakitu-gpu release: intermittently fails GCETest
      • 798540: jetstream_ApiServerDeveloperConfiguration flake: Timed out waiting for AP to appear operational
      Resolved Issues:
      • 798558factory-strago-7458.B build failure since 7458.357.0
      • 798618: veyron_minnie-paladin builds fail with cheets_StartAndroid.stress
      • 798649: chromite.scripts.merge_logs_unittest failed due to acrossing year
      • 798273: HWTest bvt-arc aborted cheets_CTS_N* and cheets_GTS* tests stuck on "waiting for cache lock"
      • 797620Build failure chrome-pfq: login_CryptohomeIncognito and security_ProfilePermissions.guest are failing across multiple boards

      12/25-12/29
      Sheriffs: jcliang, semenzato, renyi

      Ongoing Issues:
      • 783832: cheets_StartAndroid.stress fails on release builders and CQ.  Trying to reproduce.
      • 797620Build failure chrome-pfq: login_CryptohomeIncognito and security_ProfilePermissions.guest are failing across multiple boards
      • 796254: Archive stage fails in build_image
      • 796275bvt-arc times out across boards
      • 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed - seen across many canaries.
      • 795128: evemu-device did not die resulting in security_SandboxedServices failing
      • 794242: chromeos6-row22-jetstream-host5 repeatedly failing tests
      • 796684: coral-paladin can't finish very often
      • 796737: coral-release hasn't succeeded since build 599
      • 782832: not enough daisy_skate devices to keep bvt pool alive
      • 795912: kefka-release:1758-1773 failed
      • 784914: DUT cannot reboot at pre-setup of rootfs update
      • 715011: nvmem ec test crashes
      • 795902: fizz-release failing since Dec 8
      Resolved Issues:
      • 797314: cheets_MediaPlayerVideoHWDecodeUsed failing across boards
      • 797599Build failure on coral-release builder: build_packages failed

      12/18-12/22
      Sheriffs: bmgordon, martinroth, chenghan

      Ongoing issues:
      • 797314: cheets_MediaPlayerVideoHWDecodeUsed failing across boards
      • 796254: Archive stage fails in build_image
      • 796275bvt-arc times out across boards
      • 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed - seen across many canaries.
      • 795128: evemu-device did not die resulting in security_SandboxedServices failing
      • 794242: chromeos6-row22-jetstream-host5 repeatedly failing tests
      • 796684: coral-paladin can't finish very often
      • 796737: coral-release hasn't succeeded since build 599
      • 782832: not enough daisy_skate devices to keep bvt pool alive
      • 795912: kefka-release:1758-1773 failed
      • 784914: DUT cannot reboot at pre-setup of rootfs update
      • 715011: nvmem ec test crashes
      • 795902: fizz-release failing since Dec 8
      Resolved issues:
      • 796212: graphics_Idle: Unhandled ZeroDivisionError
      • 796916: reef unibuild config doesn't validate

      12/11-12/15
      Sheriffs: dtor, ecgh, deanliao

      12/04-12/08
      Sheriffs: hungte, rspangler, caveh

      Ongoing issues:
      • 767953: cheets_StartAndroid.stress: FAIL: Android did not boot! (first reported 22-Sep; maybe recurring now)
      • 789077: release: RootfsUpdateError: Update failed with unexpected update status: UPDATE_STATUS_IDLE'
      • 792262: Chrome Pre-Flight Exceptions on M64 Branch
      • 792592: ap-demons unit test failing with dbus errors on several canaries
      • 792667: CQ failure: Moblab AFE timeout
      • 793356: peach-pit-chrome-pfq failed HWTest because of no DUTs
      • 793447: M63 builders failing with INVALID_BUILD_DEFINITION on stabilize branch
      • 793499: Hwtest provision error on several chrome PFQs and informational PFQs
      Resolved Issues:
      • 791600: Master scheduler is down
      • 791786caroline-tot-chrome-pfq-informational failed HWTest security_OpenFDs
      • 791916: Master scheduler down with NoHostIdError
      • 792115 -> 791643: TestSimpleChromeWorkflow stage failing due to gsutil creds not updating
      • 792565desktopui_ScreenLocker failing on betty (--> removed from smoke test)
      • 792753: chromeos-firmware-coral build issues
      • 792985: CQ failure: MySQL Cannot execute statement
      • 757625: smbprovider unit tests failing ASAN builds
      Misc:
      • 792536: Need coral testing for branch builds (--> just turned on; may result in new bugs)


      11/27-12/01
      Sheriffs: drinkcat, athilenius, slavamn

      Ongoing issues

      Needs attention:
      Assigned but not fixed (?):
      • 789062: guado_moblab-paladin failed due to "lxc-clone: command not found"
      • 789451novato-arc64-release: Target image has run out of space
      • 740408: sheriffing rotation: No sheriff displayed on Monday morning TPE time
        • Patch needs OWNERS review
      • 788628: HWTest bvt-arc keeps timing out on a few boards
      Deputy stuff:
      • 784914: provision failurs: DUT cannot reboot at pre-setup of rootfs update
        • 788584kefka/coral-release/paladin: Linksys USB3GIGV1 Ethernet adapter fails to enumerate (r8152, usb X-Y: device not accepting address Z, error -62)
        • 788589: kefka-release: cannot recover from reboot at post check of stateful update // pre-setup of rootfs update  (duped to above)
      • Missing DUT sadness:
        • 782832not enough daisy_skate devices to keep bvt pool alive
        • 788586: daisy_spring: Not enough DUTs for board: pool: bvt; required: 4, found: 3
        • 788596: veyron_rialto: No good devices in pool:bvt
        • 780738: M64: FAIL builds of veyron_tiger since 10/28
        • 789352enguarde: Not enough DUTs for board: enguarde, pool: bvt; required: 4, found: 2
        • 789420: pyro-release: bvt-arc suite timeout
      Resolved issues
      • 788455: lxc-start failing in HWTest for electro and basking
        • 788595: pyro-release: lxc-start failing in HWTest for pyro
        • ultima-release as well
      • 788925: File dir-ROOT-A/opt/google/chrome/libwidevinecdm.so contains unsatisfied symbols: set(['\x07\x01'])
        • Reverted libwidevinecdm change, hmchen and xhwang are looking
        • AI: Could we possibly run ImageTest in Chrome PFQ to avoid this issue next time?
      • 789839: chromium-pfq: BuildPackages: chromeos-chrome: Command 'lsb_release -a' returned non-zero exit status 3
        • Broke -master and pfq for a few builds...
      Misc:
      • 789461: eve-release: cheets_ContainerMount: Mount points are mismatched with the expected list
      • 788017: falco-release times-out at BuildPackage
      • 788592: nefario-release: The BuildPackages [afdo_use] stage failed: Packages failed in ./build_packages: sys-boot/depthcharge
      Flakes and other issues (not fixed but not consistently failing either):
      • 789077: -release: RootfsUpdateError: Update failed with unexpected update status: UPDATE_STATUS_IDLE'
      • 788591: mccloud: graphics_GLMark2: crash in i915_gem_retire_requests_ring/i915_gem_object_move_to_inactive

        11/13-11/17
        Sheriffs: benchan, nsanders, hiroh

        Ongoing issues
        • 784462Provision failure spike in the lab
          • (Duplicated) 784222: PaygenTestDev failed on multiple canary builds
        • 784225: TestLabException: Not enough DUTs on Chrome-PFQ, Android-PFQ and canary build
        • 784686: veyron_rialto-paladin failed at BuildImage staging due to package: chromeos-base/telemetry
        • 786159: ImportError: No module named lockfile
        • 786159: HWTest failed due to INVALID_OPTIONS
        • 786159: AFE is down: google-sso enforced a new config requirement, breaking our apache servers
        • 786167: auto-update failed with StatefulUpdateError
        • 786395: CQ master failed to push a change with 'git log' errors
        • 786487: reef-uni-paladin failed due to no valid hosts for board:reef-uni
        • 785552: provision failures: DUT cannot recover from reboot at post check of rootfs update

        11/6-11/10
        Sheriffs: puthik, ddavenport, cywang

        Resolved issues
        • 782509video_ChromeHWDecodeUsed mse tests are failed because crosvideo.appspot.com is broken down.
        • 781845: desktopui_ScreenLocker failing on amd64-generic and betty
        • 781302: slow queries on shards | chromeos-server98 and 104 tick rate is really low
        • 783312: video_ChromeHWDecodeUsed failing on tricky, caroline, lumpy, peppy
        • 781852: CQ failure when there are no CLs in the CQ run
        • 783449: unittest flake in autotest_lib.site_utils.lxc.container_pool.client_unittest.ClientTests.testConnection
        Ongoing issues
        • 776997: cheets_StartAndroid.stress failes and chrome / kernel crashes
        • 783832: cheets_StartAndroid.stress timeout

        10/30-11/6
        Sheriffs: teravest, justincarlson, cywang
        • 782509: widespread Media.GpuVideoDecoderInitializeStatus not loaded or histogram bucket not found or histogram bucket found at < 100%" - the root cause is "404 in crosvideo.appspot.com". hiroh@ is helping to make a workaround to redirect requests to crosvideo2.appspot.com temporarily.
        • 782577incorrect dependencies of media-libs/arc-camera3-libcamera_jpeg (Fixed)

        10/30-11/3
        Sheriffs: teravest, justincarlson, fukino
        • 777920[kernel 3.18] veyron_speedy provision failure: USB enumeration of ethernet adapter fails with "can't set config #1, error -71"
        • 768542: DUT fails to bring up USB ethernet adapter after reboot in provision (chromeos kernel 4.4)
        • 779583: General Protection Fault in kernel-list_move_tail called from i915
          • Causes graphics_Idle failures
        • 780515: daisy_skate-release:1910 failed
          • Paygen failures
        • 780045: BuildPackages failing to build chromeos-chrome
          • This should be resolved, but keep an eye on the next goma update.
        • 780503: cave-release:1635 failed
        • 765686: wizpig-paladin Provision failed: Post-provision check for "system-services" being "start/running" can fail
          • This needs more attention and debugging.

        10/23-10/27
        Sheriffs: akahuang, jinsong, mruthven
        • 777250HWTest failed to provision on peach_pit and veyron_minnie, let Chrome gardener to triage
        • 776919: lakitu-gpu, lakitu, lakitu paladin failed at build_package, should be fixed by CL:735061 and CL:737773
        • 766259: buildstart stage failing with IntegrityError, a flaky failure.
        • 777829: Most paladins raised exception "process killed by signal 9"

          10/16-10/20
          Sheriffs: groeck, xiaochu, fukino, tetsui
          • 775872: M64: Cyan, Eve, Kefka, Samus build is RED for 4 days

          10/9-10/13
          Sheriffs: jclinton, furquan, posciak
          • 773185: All Chrome PFQ bots failing starting from 63.0.3237.0 due to a syntax error in DEPS
          • 772568: lumpy, peppy, tricky Chrome PFQ failures in vmtest; manual uprev via 773446

          10/2-10/8
          Sheriffs: ntang, djkurtz, phobbs
          • 771396: Lab DNS failure caused wide spread master-paladin filaure.
          • 771236: Provision failure due to version '9999'
          • 772582: Puppet run may interrupt the ssh_config and causes ssh conntection failure.
          • 770778: A few cases of shard apache process death, which needs alerting.
          • 770865: Shard db inconsistent with master db causes shard_client crashloop
          • 770715:  Quite a few graphics_drm failure (fixed).
            9/25-10/1
            Sheriffs: chinyue, vbendeb, mxt
            • 769099autotest-server & autotest-web-frontend circular dep
            • 769334betty-arc64-paladin failed VMTest
            • 768280: build_image run out of space

              9/18-9/24
              Sheriffs: puneetster, amstan, 

              OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)
              Comments