There are two distinct pieces that often get called the same thing:
Telemetry-based scrolling / performance benchmarks -- A thin Python wrapper around Telemetry that uses it to open Chrome with a special flag (
--enable-gpu-benchmarking), loads a bunch of web pages in succession, and for each of those web pages executes some JS that calls private window.chrome APIs that are only exposed when
--enabled-gpu-benchmarking is on. These APIs then return rendering stats (how much time we spent painting, how many composites have occurred over the lifetime of the page, etc) to JS, and then Telemetry tunnels them out to the scrolling benchmark (i.e. back in Python land). The scrolling benchmark then saves them or prints to console. Read more about these telemetry benchmarks.
Great overview of Telemetry - deck
from Chrome Testing Summit.
- Write one performance test that can trivially run on all platforms [Mac/Win/Linux, CrOS, Android] for both Chrome and ContentShell.
- Run on browser binaries, without having to build the browser yourself.
- Run on non-chrome browsers for comparative studies.
- Use WebPageReplay to get repeatable test results.
Should my test use Telemetry?
Well, it depends on what you are trying to test or measure.
Telemetry is designed for performance tests that:
Are you trying to verify correctness? If yes, Telemetry is not for you but browser tests are your friend.
- Can be written with pure JS/HTML/CSS
- Use real world content (rather than synthetic)
- Could be run on other browsers, not just Chrome
Setting Up Telemetry (instructions per platform)
Windows/Mac/Linux: You need a Chrome binary (any will do, including standard official release builds or Canaries), Python, and a Chrome source checkout (you just need the checkout, you don't have to build it! - get the code). In order to use cached sites (webpage replay), remember to download src-internal (http://go/read-src-internal/).
Android: You need ADB, an Android phone with a recent version of Chrome for Android set up for remote debugging (no root required!), and a Chrome checkout set up to build for Android. Follow these steps to get up and running:
- Configure your Chrome checkout to build for Android by following the "Getting the code" steps on AndroidBuildInstructions (follow exactly, in order).
- You need to have ADB installed and on your host machine's PATH.
- On Linux: there's a version of ADB already checked into the Chrome source and Telemetry will use it, so you don't need to do anything.
- On Mac: you can get ADB from the Android developer's website SDK download page, either in the full ADT bundle or just the SDK tools. In either case, make sure the platform-tools directory is in your PATH.
- On Windows: Telemetry for Android doesn't work yet, but probably just requires a couple simple hacks to get working.
- Build the forwarder. (Note that crbug.com/169706 will eliminate this step)
ninja -C out/Release forwarder2 md5sum
- Next, enable USB debugging on your phone. Instructions again available on the Android developer's website under Using Hardware Devices.
- Last, if your phone is not rooted, enable remote USB debugging in the Chrome for Android app. Note you do not need to issue any ADB commands (e.g. TCP port forwarding on that page), you just need to turn on USB Web Debugging for Chrome under Settings > Advanced > Developer tools. If you have either "adb root" or "adb shell su", you can skip this step.
Now if you plug your phone into your desktop and run:
you should see Chrome for Android options. Once you see that, you're all set, and can continue with "Running a Benchmark" below.
- You will need to run telemetry from another machine - follow Windows/Mac/Linux instructions above.
- Ensure that your ChromeOS device has a test image on in it. If it doesn't, download one from go/chromeos-images.
- Enable dev mode and boot from USB on your Chromebook - device specific instructions
- Instructions for running a test image
- Make sure that your chromebook and the machine with the chrome checkout are on the same network. Typically, you'll connect both to Google's network via ethernet.
- Get the IP address of your chromebook
- Ctrl+alt+F2 gets you to to a bash prompt
- sign in as chronos
- enter sudo su
- enter ifconfig
- grab inet addr: in the eth0 section
- Set up password-less ssh to your test machine
- cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa ~/.ssh/
- cp /chromium/src/third_party/chromite/ssh_keys/testing_rsa.pub ~/.ssh/
- Add the following lines to .ssh/config
- Host <chromebook ip address>
- CheckHostIP no
- StrictHostKeyChecking no
- IdentityFile %d/.ssh/testing_rsa
- Protocol 2
- Run telemetry with the following options:
- --remote=<ip address>
Running a Benchmark
To run a benchmark, use the
run_multipage_benchmarks script. For example, to run the scrolling benchmark over the top 25 pages:
tools/perf/run_multipage_benchmarks --browser=canary smoothness_measurement tools/perf/page_sets/top_25.json
Lets break down this command a bit:
- tools/perf is where we keep our benchmarks that are written in Python.
- run_multipage_benchmarks is the script we use to run a benchmark across a list of pages
- --browser=canary tells the script to use Chrome Canary, if it is installed on the system. If you dont have canary [eg you're on linux] it'll fail and tell you to give it another browser.
- --browser=list for all browsers that the script thinks it can use. Pass --browser-list -vvv if you're not seeing a browser you expect to see.
- --browser=system: the stable chrome install on your system
- --browser=debug or release: chromium from out/Debug etc, if it was found
- --browser=content-shell-debug: a content shell build found in out/Debug
- --browser=android-chrome: chrome detected on an attached android device via adb
- --browser=cros-chrome --remote=$CHROMEBOOK_IP: chrome running on your chromebook
- --browser=exact --exact-browser=<path to build>: your tests will work with any chrome build >= M18!
- smoothness_measurement is the name of the benchmark to run. If you type
./run_multipage_benchmarks --browser=system, you'll see a list of other benchmarks that we support. There are a lot, from JSGameBench, to Dromao. Smoothness is our catch all test for graphics.
- tools/perf/page_sets/top_25.json is a list of 25 pages that we monitor continuously on our bots. The benchmark you pick will run on these pages. There are other pages, for example "key_desktop_sites" and "key_mobile_sites" as well as "tough_scrolling_cases." Some have hundreds or thousands of sites. Some have only a few. Pick the one that fits your goal.
By default, telemetry starts the browser with the default arguments in a completely clean profile. To change the arguments that chrome is launched with, pass
--extra-browse-args="--arg1 --arg2" For example, to run a benchmark in forced, threaded compositing mode:
tools/perf/run_multipage_benchmarks --browser=system smoothness_benchmark tools/perf/page_sets/top_25.json --extra-browser-args="--force-compositing-mode --enable-threaded-compositing"
Run telemetry with --browser=cros-chrome and --remote=<ip address>
While running a benchmark, do not disconnect your device. This can leave your Chrome for Android unable to reach the internet.
If this happens, run the appropriate command below (for Chrome or content shell):
line (no tmp/)
Recording a Page Set
Each page_set should have a recorded web page archive associated with it. This allows the benchmark to be run stably over the same web content. To record a new archive:
tools/perf/record_wpr --browser=(release|system) page_set
tools/perf/record_wpr --browser=system tools/perf/page_sets/top_25.json
To record only some pages in the page set, use --page-filter, for example:
tools/perf/record_wpr --browser=system --page-filter=wikipedia tools/perf/page_sets/top_25.json
List of benchmarks and page sets: tools/perf/run_multipage_benchmarks --browser=system
List of valid browser options: tools/perf/run_multipage_benchmarks --browser=list
Code and Architecture
The framework lives in
src/tools/telemetry and the benchmarks in
On top of Telemetry, there are two fundamental concepts:
-- benchmark: a python script that instrument and collect data, these live in src/tools/perf/perf_tools
-- page set: a collection of pages (either links for live sites, or a WebPageReplay archive) against which we run the benchmark.
- Create a new file in
page_benchmark and create a
PageBenchmark object. At a minimum, it must implement the
MeasurePage() method and call
results.Add(name, units, value). The
For example, the following benchmark returns "4":
from telemetry.page import page_benchmark
def MeasurePage(self, page, tab, results):
results.Add('four', 'number', four)
- See "Running a Benchmark" above to run the new benchmark.
- To make the perf bots run your new benchmark, see Adding Performance Tests.
To customize the browser command line, implement
. For example:
def CustomizeBrowserOptions(self, options):
To report the value of a UMA histogram, call
def CustomizeBrowserOptions(self, options):
def MeasurePage(self, page, tab, results):
results.Add('result', 'units', result)
To report the value of any events in the Inspector's timeline, call tab.timeline_model.GetAllEvents(). See loading_benchmark.py for a good example of this.
Doing It All By Hand
Benchmarks and page_sets are the best way to use telemetry. But, if you just want to print the number of divs on google.com, for example:
options = telemetry.BrowserOptions()
parser = options.CreateParser('telemetry_perf_test.py')
options, args = parser.parse_args(args)
browser_to_create = telemetry.FindBrowser(options)
with browser_to_create.Create() as b:
Contact Us or Follow Along
You can keep up with Telemetry related discussions and code reviews by joining the telemetry group