The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.
For Developers‎ > ‎

Telemetry: Performance testing framework


Telemetry 101

There are two distinct pieces that often get called the same thing:

Telemetry is moving super fast and has become the solution for cross-(chrome)-platform performance testing, with more aspects being added at a furious pace. It originally started by tonyg@ and nduca@ to measure scrolling performance, and is now used to measure memory, javascript benchmarks, page cyclers, and more every day. Telemetry is essentially a python wrapper on top of the "DevTools Remote Debugging Protocol", and it also abstracts launching / connecting / opening a page in whatever underlying platform chrome is running, fetching data via Inspector timeline, traces, etc. It knows how to start Chrome with arbitrary flags and knows how to use Web Page Replay to load pages from disk instead of network.

Telemetry-based scrolling / performance benchmarks -- A thin Python wrapper around Telemetry that uses it to open Chrome with a special flag (--enable-gpu-benchmarking), loads a bunch of web pages in succession, and for each of those web pages executes some JS that calls private window.chrome APIs that are only exposed when --enabled-gpu-benchmarking is on. These APIs then return rendering stats (how much time we spent painting, how many composites have occurred over the lifetime of the page, etc) to JS, and then Telemetry tunnels them out to the scrolling benchmark (i.e. back in Python land). The scrolling benchmark then saves them or prints to console. Read more about these telemetry benchmarks.

Great overview of Telemetry - deck from Chrome Testing Summit.

How to write benchmarks in telemetry: Benchmarks == Measurements + Page Set

Goals

  • Write one performance test that can trivially run on all platforms [Mac/Win/Linux, CrOS, Android] for both Chrome and ContentShell.
  • Run on browser binaries, without having to build the browser yourself.
  • Run on non-chrome browsers for comparative studies.
  • Use WebPageReplay to get repeatable test results.

Should my test use Telemetry?

Well, it depends on what you are trying to test or measure.
Telemetry is designed for performance tests that:
  • Can be written with pure JS/HTML/CSS
  • Use real world content (rather than synthetic)
  • Could be run on other browsers, not just Chrome
Are you trying to verify correctness?  If yes, Telemetry is not for you but browser tests are your friend.

Useful Links

Using Telemetry

Setting Up Telemetry (instructions per platform)

Windows/Mac/Linux: You need a Chrome binary (any will do, including standard official release builds or Canaries), Python, and a Chrome source checkout (you just need the checkout, you don't have to build it! - get the code). In order to use cached sites (webpage replay), remember to download src-internal (http://go/read-src-internal/). 

Android: You need ADB, an Android phone with a recent version of Chrome for Android set up for remote debugging (no root required!), and a Chrome checkout set up to build for Android. Follow these steps to get up and running:

  1. Configure your Chrome checkout to build for Android by following the "Getting the code" steps on AndroidBuildInstructions (follow exactly, in order).
  2. You need to have ADB installed and on your host machine's PATH.
    1. On Linux: there's a version of ADB already checked into the Chrome source and Telemetry will use it, so you don't need to do anything.
    2. On Mac: you can get ADB from the Android developer's website SDK download page, either in the full ADT bundle or just the SDK tools. In either case, make sure the platform-tools directory is in your PATH.
    3. On Windows: Telemetry for Android doesn't work yet, but probably just requires a couple simple hacks to get working.
  3. Build the forwarder. (Note that crbug.com/169706 will eliminate this step)
    ninja -C out/Release forwarder2 md5sum
  4. Next, enable USB debugging on your phone. Instructions again available on the Android developer's website under Using Hardware Devices.
  5. Last, if your phone is not rooted, enable remote USB debugging in the Chrome for Android app. Note you do not need to issue any ADB commands (e.g. TCP port forwarding on that page), you just need to turn on USB Web Debugging for Chrome under Settings > Advanced > Developer tools. If you have either "adb root" or "adb shell su", you can skip this step.
Now if you plug your phone into your desktop and run:

tools/perf/run_multipage_benchmarks --browser=list

you should see Chrome for Android options. Once you see that, you're all set, and can continue with "Running a Benchmark" below.

Chrome OS: 

  1. You will need to run telemetry from another machine - follow Windows/Mac/Linux instructions above.
  2. Ensure that your ChromeOS device has a test image on in it. If it doesn't, download one from go/chromeos-images.
    1. Enable dev mode and boot from USB on your Chromebook - device specific instructions
    2. Instructions for running a test image
  3. Make sure that your chromebook and the machine with the chrome checkout are on the same network. Typically, you'll connect both to Google's network via ethernet. 
  4. Get the IP address of your chromebook
    1. Ctrl+alt+F2  gets you to to a bash prompt 
    2. sign in as chronos
    3. enter sudo su
    4. enter ifconfig
    5. grab inet addr: in the eth0 section
  5. Set up password-less ssh to your test machine
    1. cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa ~/.ssh/
    2. cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa.pub ~/.ssh/
    3. Add the following lines to .ssh/config
      1. Host <chromebook ip address> 
      2. CheckHostIP no 
      3. StrictHostKeyChecking no 
      4. IdentityFile %d/.ssh/testing_rsa 
      5. Protocol 2
  6. Run telemetry with the following options:
    1. --browser=cros-chrome
    2. --remote=<ip address>

Running a Benchmark

To run a benchmark, use the run_multipage_benchmarks script. For example, to run the scrolling benchmark over the top 25 pages:

tools/perf/run_multipage_benchmarks --browser=canary smoothness_measurement tools/perf/page_sets/top_25.json

Lets break down this command a bit:
  • tools/perf is where we keep our benchmarks that are written in Python.
  • run_multipage_benchmarks is the script we use to run a benchmark across a list of pages
  • --browser=canary tells the script to use Chrome Canary, if it is installed on the system. If you dont have canary [eg you're on linux] it'll fail and tell you to give it another browser.
    • --browser=list for all browsers that the script thinks it can use. Pass --browser-list -vvv if you're not seeing a browser you expect to see.
    • --browser=system: the stable chrome install on your system
    • --browser=debug or release: chromium from out/Debug etc, if it was found
    • --browser=content-shell-debug: a content shell build found in out/Debug
    • --browser=android-chrome: chrome detected on an attached android device via adb
    • --browser=cros-chrome --remote=$CHROMEBOOK_IP: chrome running on your chromebook
    • --browser=exact --exact-browser=<path to build>: your tests will work with any chrome build >= M18!
  • smoothness_measurement is the name of the benchmark to run. If you type ./run_multipage_benchmarks --browser=system, you'll see a list of other benchmarks that we support. There are a lot, from JSGameBench, to Dromao. Smoothness is our catch all test for graphics.
  • tools/perf/page_sets/top_25.json is a list of 25 pages that we monitor continuously on our bots. The benchmark you pick will run on these pages. There are other pages, for example "key_desktop_sites" and "key_mobile_sites" as well as "tough_scrolling_cases." Some have hundreds or thousands of sites. Some have only a few. Pick the one that fits your goal.

By default, telemetry starts the browser with the default arguments in a completely clean profile. To change the arguments that chrome is launched with, pass  --extra-browse-args="--arg1 --arg2" For example, to run a benchmark in forced, threaded compositing mode:

tools/perf/run_multipage_benchmarks --browser=system smoothness_benchmark tools/perf/page_sets/top_25.json --extra-browser-args="--force-compositing-mode --enable-threaded-compositing"

ChromeOS

Run telemetry with --browser=cros-chrome and --remote=<ip address>

Android

While running a benchmark, do not disconnect your device.  This can leave your Chrome for Android unable to reach the internet.
If this happens, run the appropriate command below (for Chrome or content shell):

/data/local/chrome-command-line (no tmp/)

/data/local/tmp/content-shell-command-line

Recording a Page Set

Each page_set should have a recorded web page archive associated with it. This allows the benchmark to be run stably over the same web content. To record a new archive:

tools/perf/record_wpr --browser=(release|system) page_set

For example:

tools/perf/record_wpr --browser=system tools/perf/page_sets/top_25.json

To record only some pages in the page set, use --page-filter, for example:

tools/perf/record_wpr --browser=system --page-filter=wikipedia tools/perf/page_sets/top_25.json

Useful/Common Commands

List of benchmarks and page sets: tools/perf/run_multipage_benchmarks --browser=system
List of valid browser options: tools/perf/run_multipage_benchmarks --browser=list

Modifying Telemetry

Code and Architecture

The framework lives in src/tools/telemetry and the benchmarks in src/tools/perf.

On top of Telemetry, there are two fundamental concepts:
-- benchmark: a python script that instrument and collect data, these live in src/tools/perf/perf_tools
-- page set: a collection of pages (either links for live sites, or a WebPageReplay archive) against which we run the benchmark.

Adding a Benchmark

  1. Create a new file in src/tools/perf/perf_tools.
  2. Import page_benchmark and create a PageBenchmark object. At a minimum, it must implement the MeasurePage() method and call results.Add(name, units, value). The tab.EvaluateJavaScript() method is useful for interacting with the page.

    For example, the following benchmark returns "4":

    from telemetry.page import page_benchmark

    class AdditionBenchmark(page_benchmark.PageBenchmark):
      def MeasurePage(self, page, tab, results):
        four = tab.EvaluateJavaScript('2+2')
        results.Add('four', 'number', four)

  3. See "Running a Benchmark" above to run the new benchmark.
  4. To make the perf bots run your new benchmark, see Adding Performance Tests.
To customize the browser command line, implement CustomizeBrowserOptions(). For example:

def CustomizeBrowserOptions(self, options):
  options.AppendExtraBrowserArg('--my-lovely-command-line-arg')

To report the value of a UMA histogram, call domAutomationController.getHistogram() from JavaScript. For example:

def CustomizeBrowserOptions(self, options):
  options.AppendExtraBrowserArg('--dom-automation')
  options.AppendExtraBrowserArg('--reduce-security-for-dom-automation-tests')

def MeasurePage(self, page, tab, results):
  result = tab.EvaluateJavaScript('domAutomationController.getHistogram("MY_LOVELY_HISTOGRAM")')
  results.Add('result', 'units', result)

To report the value of any events in the Inspector's timeline, call tab.timeline_model.GetAllEvents(). See loading_benchmark.py for a good example of this.

Doing It All By Hand

Benchmarks and page_sets are the best way to use telemetry. But, if you just want to print the number of divs on google.com, for example:

import telemetry
options = telemetry.BrowserOptions()
parser = options.CreateParser('telemetry_perf_test.py')
options, args = parser.parse_args(args)
browser_to_create = telemetry.FindBrowser(options)
with browser_to_create.Create() as b:
  b.tabs[0].Navigate('http://www.google.com')
  print b.tabs[0].EvaluateJavaScript('document.querySelectorAll("div").length')

Contact Us or Follow Along

If you have questions, please email telemetry@chromium.org
You can keep up with Telemetry related discussions and code reviews by joining the telemetry group.

Comments