Telemetry: Unittests

Telemetry has two suites of unittests:

  1. tools/telemetry/run_tests # Verifies the core framework functions properly
  2. tools/perf/run_tests # Verifies the benchmarks that use the framework function properly

These are functional tests that do not depend on performance.


Triaging failures


  • Is there a native stack? Since these tests interact with a lot of recorded real world content, they unintentionally end up serving as integration tests which frequently uncover chromium crashes. If you see a native crash stack (after a TabCrashException or BrowserGoneException), this is guaranteed to be a chrome issue. Usually scanning the change log for patches that touch files that show up in the stack will point to the culprit to revert.
  • Is there a python stack? If there's a python-only exception, it is very likely, but not guarantee to be a Telemetry breakage. Look for Telemetry changes in the range for a culprit.
  • Is there a timeout? These could go either way and are tricky to diagnose, move on to local diagnostics.


Diagnosing failures locally

  • Identify the failing test.
  • Build a version of chrome in which it fails (change your gyp file to switch e.g. from ChromeOS to Chrome)
  • Run test via:

    $ tools/telemetry/run_tests <test>

     whereas <test> can be e.g.

          BrowserTest.testForegroundTab

     or

          BrowserTest

     as a “wildcard” by matching the sub string "BrowserTest".


and see that it fails for you as well. If not, you might want to try some of these flags:


--browser=list       # shows all browsers (debug / release).

--browser=<version> # By default the last compiled browser will be taken, but "Debug" or "Release" can be specified to override.

--repeat=<N>         # repeats the test N times. E.g. --repeat=5. Note that flaky tests might fail after repetition.


  • Analyze what is going on by adding log statements and add the option "--show-stdout" to see them

Here are some other flags which might come handy:


--show-stdout            # shows the LOG statements

--extra-browser-args=..  # Let’s guess what that might be :)


The usage of GDB would need to be possibly done by changing the scripts.


Disabling tests

Tests should generally only be disabled for flakiness. Consistent failures should be diagnosed and the culprit reverted.

The @test.Disabled and @test.Enabled decorators may be added above any test to enable or disable it. They optionally accept a list of platforms, os versions or browser types. Examples:
  • @test.Disabled # Disabled everywhere
  • @test.Enabled('mac') # Only runs on mac
  • @test.Disabled('xp') # Run everywhere except windows xp
  • @test.Disabled('debug') # Run everywhere except debug builds
Comments