The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.

Telemetry: Run Benchmarks Locally

Telemetry: Run Benchmarks Locally

back to Telemetry Introduction



Setting Up Your Environment

To run a Telemetry test on any platform you will need a Chrome checkout on Linux, Mac, or Windows*.  Follow these instructions to get the code.  To run benchmarks using cached sites (highly recommended) you will need to also download src-internal (work in progress to move files out of src-internal).


You are now all set to run Telemetry benchmarks on Linux, Mac, and Windows!  Jump to Commands to Run Benchmarks for further instruction.


Android

*To run a Telemetry test on Android, you need a Linux host machine (Windows and Mac OSX not yet supported) and Chrome for Android installed.  Follow these steps to get setup:


  1. You need to have ADB on your path. If you've checked out chromium with "target_os = ['android']" in your .gclient file there’s a version already checked into the chromium source so you don’t have to do anything!

  2. Enable USB debugging on your device.

  3. Run ADB as root or have the SU command installed on a device that provides root privileges. Running ADB as root is preferable, you would need a "userdebug" build of Android running on your device. To make ADB run as root, on the host: $ adb root Sometines (when?) is is also required to remount the /system partition in R/W mode, to do that, on the host: $ adb remount


Plug your phone into your Linux host machine and run:

$ tools/perf/run_measurement --browser=list

If you get Chrome for Android options, you’re all set!  Continue with Commands to Run Benchmarks.


Chrome OS

To run a Telemetry test on Chrome OS, you will need a Linux, Mac, or Windows host machine.  Follow these steps to get setup:


  1. You must have a test image on your Chrome OS device.

    • Determine your device code name here.

    • Download a test image for that code name here.

    • Unzip the download file and copy chromiumos_test_image.bin to a USB stick. Be sure to replace /dev/sdc with the actual device of your USB stick (can be determined by running lsblk). $ sudo dd if=path/to/chromiumos_test_image.bin of=/dev/sdc bs=4M oflag=sync

    • Enable dev mode and boot from USB on your Chromebook - device specific instructions

    • Insert the USB stick into the device, reboot, and press Ctrl+U to boot from the USB

    • After booting, press Ctrl+Alt+F2 to enter a command prompt and run /usr/sbin/chromeos-install.

  2. Connect your Chromebook and your host machine to the same network

  3. Get the IP address of your chromebook

    • Ctrl+alt+F2 will get you to a bash prompt

    • sign in as chronos

    • enter command  sudo su

    • enter command ifconfig

    • you want the address for inet: in the en0 section

  4. Set up a password-less ssh to your test machine

$ cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa .ssh/

$ cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa.pub .ssh/

  1. Add the following lines to .ssh/config

Host <chromebook ip address>

CheckHostIP no

StrictHostKeyChecking no

IdentityFile %d/.ssh/testing_rsa

Protocol 2

  1. Use the following options when running Telemetry: --browser=cros-chrome and --remote=<ip address>

Run Benchmark or Run Measurement

The performance bots use the run_benchmark script to simplify running the same benchmarks across all platforms. A benchmark is a grouping of a measurement, a page set, and configuration options. Use ./run_benchmark list for a list of all of the benchmarks. To run a benchmark with the same page set and options as the performance bots, use the run_benchmark script.

If you want to run a measurement with a different page set than the bot or with a different browser option (say --force-compositing-mode) then you should use run_measurement.

Commands to Run Benchmarks

Example Command

To run a benchmark, we use the run_benchmark script.  Here is an example command:

$ tools/perf/run_benchmark --browser=canary smoothness.top_25


List Benchmarks

To get a list of the available benchmarks, use the following command:
$ tools/perf/run_benchmark list

Extra Options

For a list of supported options, use the following command:

$ tools/perf/run_benchmark --help

Commands and Options to Run Measurements

Example Command

To run a measurement, we use the run_measurement script.  Here is an example command:

$ tools/perf/run_measurement --browser=canary smoothness tools/perf/page_sets/top_25.py


Extra Options

For a list of supported options, use the following command:

$ tools/perf/run_measurement --help


A few especially useful options:

--output-format=OUTPUT_FORMAT options can be buildbot, block, csv

--browser=cros-chrome and --remote=<ip address> for running on ChromeOS


Command Details

Let’s break down the example command:

tools/perf/run_measurement is the script we use to run a measurement across a list of pages


--browser=canary tells the script to use Chrome Canary.  Use --browser=list for all of the browsers that the script thinks it can use.  Telemetry tests will work with any Chrome build >= M18 with --browser=exact --browser-executable=<path to build>.


smoothness is the name of the measurement to run.  If you type ./run_measurement you’ll see a list of the measurements that we support.


tools/perf/page_sets/top_25.py is the path to the page set you want. When you run a benchmark you do not need to specify a page set. A benchmark is defined by a measurement and a page set. Measurements, however, require specifying a page set.



Repro-ing failures seen on Telemetry bots


Let's say you are seeing failures on the Linux Telemetry bots, for the media measurement.


You can try to repro the errors locally, using either reference builds that are under $src/chrome/tools/test/reference_build or by using update_reference_build.py to get builds for a specific version.



To run the media measurement against a reference build for Mac:


$src: tools/perf/run_measurement -v --output-format=buildbot

--browser=exact

--browser-executable=src/chrome/tools/test/reference_build/chrome_mac/Chromium.app/Contents/MacOS/Chromium

--output-trace-tag=_ref media_measurement

src/tools/perf/page_sets/tough_video_cases.json



To get builds for a specific revision, first figure out the revision you want. Revision info is available from the builder status page. Example for Linux: link. 


Sample sequence of steps (from here):



cd /tmp

<path to update_reference_build.py> -r 218611



This should create builds for each OS for revision 218611 under /tmp/reference_builds/reference_builds.


Then attempt to repro your failures locally, using command line similar to above, just pointing to the executable under /tmp/reference_builds/reference_builds.


Comments