Telemetry: Unittests

Telemetry has two suites of unittests:

  1. tools/telemetry/run_tests # Verifies the core framework functions properly
  2. tools/perf/run_tests # Verifies the benchmarks that use the framework function properly

These are functional tests that do not depend on performance.


Triaging failures


  • Is there a native stack? Since these tests interact with a lot of recorded real world content, they unintentionally end up serving as integration tests which frequently uncover chromium crashes. If you see a native crash stack (after a TabCrashException or BrowserGoneException), this is guaranteed to be a chrome issue. Usually scanning the change log for patches that touch files that show up in the stack will point to the culprit to revert.
  • Is there a python stack? If there's a python-only exception, it is very likely, but not guarantee to be a Telemetry breakage. Look for Telemetry changes in the range for a culprit.
  • Is there a timeout? These could go either way and are tricky to diagnose, move on to local diagnostics.


Diagnosing failures locally

  • Identify the failing test.
  • Build a version of chrome in which it fails (change your gyp file to switch e.g. from ChromeOS to Chrome)
  • Run test via:

    $ tools/telemetry/run_tests <test>

     whereas <test> can be e.g.

          BrowserTest.testForegroundTab

     or

          BrowserTest

     as a “wildcard” by matching the sub string "BrowserTest".

  • Follow the steps described in the Diagnosing Test Failures page, the command line flags listed there are accepted by run_tests and may prove useful.

The usage of GDB would need to be possibly done by changing the scripts.


Disabling tests

Tests should generally only be disabled for flakiness. Consistent failures should be diagnosed and the culprit reverted.

The @test.Disabled and @test.Enabled decorators may be added above any test to enable or disable it. They optionally accept a list of platforms, os versions or browser types. Examples:
  • @test.Disabled # Disabled everywhere
  • @test.Enabled('mac') # Only runs on mac
  • @test.Disabled('xp') # Run everywhere except windows xp
  • @test.Disabled('debug') # Run everywhere except debug builds
Comments