Telemetry: Run Benchmarks Locally

Contents

  1. 1 Set Up
    1. 1.1 Windows
    2. 1.2 Android
    3. 1.3 Chrome OS
    4. 1.4 Chrome OS (from ChromeOS src, using test_that)
  2. 2 Security Issues When Running Telemetry Remotely
  3. 3 Benchmark Commands
  4. 4 Telemetry benchmarks can be run with run_benchmark.
    1. 4.1 Running a benchmark
    2. 4.2 Running on another browser

Set Up

If you don't have a Chromium checkout, download the latest Telemetry archive. Unzip the archive. If you're running on Mac OS X, you're all set! For Windows, Linux, Android, or Chrome OS, read on.

Windows

Some benchmarks require you to have pywin32. Be sure to install a version that matches your Python version and bitness.

Linux

Telemetry on Linux tries to scan for attached Android devices with ADB. The included adb binary is 32-bit. On 64-bit machines, one needs to install the libstdc++6:i386 package.

Android

Running on Android is supported with a Linux or Mac OS X host. Windows is not yet supported. There are also a few additional steps to set up:

  1. Telemetry requires ADB. If you're running from the zip archive, ADB is already included. But if you're running with a Chromium checkout, ensure your .gclient file contains target_os = ['android'], then resync your code.

  2. If running from an OS X host, you need to run ADB as root. First, you need to install a "userdebug" build of Android on your device. Then run adb root. Sometimes you may also need to run adb remount.

  3. Enable debugging over USB on your device.

  4. You can get the name of your device with adb devices and use it with Telemetry via --device=<device_name>

Chrome OS

To run a Telemetry test on Chrome OS, you will need a Linux, Mac OS X, or Windows host machine. Follow these steps to get set up.  When setting up your DUT (Device Under Test) on a network, please pay attention to the Security Issues below.

  1. You must have a test image installed on your Chrome OS device.

  2. Connect your Chromebook and your host machine to the same network.  Please read the Security Issues section below.

  3. Get the IP address of your Chromebook.

    • Ctrl+Alt+F2 will get you to a bash prompt
    • Sign in as chronos
    • Enter command sudo su
    • Enter command ifconfig
    • You want the address for inet: in the en0 section

  4. Set up a password-less ssh to your test machine.

    $ cp src/third_party/chromite/ssh_keys/testing_rsa ~/.ssh/ $ cp src/third_party/chromite/ssh_keys/testing_rsa.pub ~/.ssh/

  5. Add the following lines to ~/.ssh/config:

    Host <chromebook ip address> CheckHostIP no StrictHostKeyChecking no IdentityFile %d/.ssh/testing_rsa Protocol 2

  6. Use the following options when running Telemetry: --browser=cros-chrome and --remote=<ip address>

Chrome OS (from ChromeOS src, using test_that)

To run a Telemetry server side test from ChromeOS src without setting up an AFE server or devserver.

  1. You must have a test setup to run autotest.

  2. Change the server side test .py file as below. args={} is the extra argument which needs to be added.

    3. def run_open(self, host=None, test=None, args={}): local = args.get("local") == "True" 

  3. Change the control file as below.

    4. from autotest_lib.client.common_lib import utils jot.run_test('telemetry_autotest', host=host, test='telemetry_test', args=utils.args_to_dict(args))

  4. Launch ChromeOS chroot pointing to Chrome source:

    $ cros_sdk  --chrome_root=<chrome_src_path>   # Full path before the src/ e.g. /local/storage/Chrome/src. The Chrome path should be /local/storage/Chrome.

  5. Run the server side autotest:

    $ test_that --args='local=True' <chromebook ip> telemetry_autotest

Security Issues When Running Telemetry Remotely

When run with the --remote option, Telemetry sets up a reverse ssh tunnel (ssh -R) from your DUT to your workstation, for the purpose of serving web pages from the Web Replay Server running on your workstation.  This tunnel lets a malicious program on your DUT, or other devices on the same network, execute arbitrary commands on your workstation.  Therefore, running Telemetry remotely in a secure fashion requires:
  1. that the DUT be connected directly to your workstation on a private network with no access to the internet;
  2. that the DUT contain a fresh image that has never been exposed to the internet;
  3. that the private network includes no other devices (except other DUTs in the same state).

Benchmark Commands

Telemetry benchmarks can be run with run_benchmark.

In the Telemetry zip archive, this located at telemetry/run_benchmark.
In the Chromium source tree, this is located at src/tools/perf/run_benchmark.

Running a benchmark

List the available benchmarks with:

$ telemetry/run_benchmark list

Here's an example for running a particular benchmark:

$ telemetry/run_benchmark --browser=canary smoothness.top_25

To see additional options, run:

$ telemetry/run_benchmark run --help

Running on another browser

To list available browsers:

$ telemetry/run_benchmark --browser=list

If you are trying to run on Android, you should see an entry similar to android-jb-system-chrome in this list. If it's not there, the device is not set up correctly.

To run a specific browser executable:

$ telemetry/run_benchmark --browser=exact --browser-executable=path_to_binary

To run on a Chromebook:

$ telemetry/run_benchmark --browser=cros-chrome --remote=ip_address
Comments