Autotest Design Patterns

Contents

1 Adding retries to Flaky tests
2 Wrapping a GTest test in Autotest
3 Wrapping a Browerstest in Autotest
4 Wrapping a Telemetry test in Autotest
5 Locally Testing a Wrapped Telemetry Test:
6 Logging into Chrome from an Autotest (using Telemetry)
7 Rebooting a machine in a server-side test
8 Running a client side test as part of a server side test
9 Writing code to automatically generate a label for a DUT
10 Use Chrome Driver in ChromeOS Autotest Test

This is a collection of patterns to help aid test developers in finding useful parts of Autotest and common pathways when writing tests.

Adding retries to Flaky tests

We have a mechanism to retry tests that have flaky failures. A retries count can be specified, and if a test fails with TestFailRetry then it will be re-attempted until it either passes or the number of failed retries reaches the retry count, in which case the test will be considered failed.

To add retries to your test, you need to do two things.

Add a RETRIES = X attribute to the control file for the test you would like to retry, where X is an integer giving the maximum number of times the test should be retried. For an example of a control file that uses retries, see https://gerrit.chromium.org/gerrit/#/c/46575/
Modify your test so that it throws a TestFailRetry exception for failures that should be retried. Any other failure exceptions will bypass the retry mechanism and cause the test to fail immediately despite retries. The simplest and most general way to do this is to catch all test exceptions and rethrow them as a TestFailRetry (see examples in https://gerrit.chromium.org/gerrit/#/c/46803/ ), but where it is possible to distinguish between failures that you want to retry and failures that you don't want to retry, a more fine grained approach is recommended.

Wrapping a GTest test in Autotest

Wrapping a Browerstest in Autotest

Wrapping a Telemetry test in Autotest

Telemetry is replacing Pyauto as the new performance testing framework for Chrome as well as the new key way to write tests that interact with the UI. Writing a test that uses telemetry is a 2 step process:

Write a telemetry test or benchmark in the chromium source.

Telemetry's source code, tests, and performance benchmarks are all platform independent. Thus they all exist within the chromium not the chromium-os source.
If one wants to write a new test, they first should add it as a new telemetry test.
They can run these tests locally against their Chrome-OS machines but provided the arguments --browser=cros-chrome and --remote=<IP> to the telemetry run_tests script that kicks off your new tests.
Once the test works and is checked in, it can be ran as a new autotest test.

It will take a day or two for the chrome tip-of-tree to be rev-ed into the next build

Please see http://dev.chromium.org/developers/telemetry for more information.

Write an autotest test to kick off the new telemetry test.

Once your new telemetry test/benchmark is checked into chrome and working you need to write an autotest server-side test to kick it off.
In your new test script you need the following code:

from autotest_lib.server.cros import telemetry_runner


def run_once(self, host=None):
        """Run the telemetry scrolling benchmark.



        @param host: host we are running telemetry on.
        """
        telemetry = telemetry_runner.TelemetryRunner(host)
        result = telemetry.run_telemetry_test('my_new_telemetry_test')


        


        # OR


        result = telemetry.run_telemetry_benchmark('my_new_telemetry_benchmark', 'my_page_set')





        # If collecting performance keyvals to be sent to the perf database:


        result = telemetry.run_telemetry_benchmark('my_new_telemetry_benchmark', 'my_page_set', keyval_writer=self)

Alternatively there currently exists a telemetry_benchmarks test and to add a new benchmark you simple need to add a new control file to this test. I.E. for octane, you add a control file named control.octane and kick off the telemetry_benchmarks test with the current parameters:

def run_benchmark(machine):
host = hosts.create_host(machine)
job.run_test("telemetry_benchmarks", host=host, benchmark="octane", page_sets=["octane.json"])

parrallel_simple(run_benchmark, machine)

Locally Testing a Wrapped Telemetry Test:

Because the telemetry runner code requires the full lab infrastructure, you need to set up a local autotest frontend and devserver in order to locally test your telemetry changes.

First set up a local AFE and database by following the instructions here: http://www.chromium.org/chromium-os/testing/autotest-developer-faq/setup-autotest-server
Kick off a local devserver.

cd src/platform/dev # From you local chromium-os checkout

# Kick off the devserver
./devserver.py --static_dir=static
Edit your local AFE to use your local devserver.

Add the following to src/third_party/autotest/files/shadow_config.ini:

[CROS]
dev_server: http://[YOUR IP ADDRESS]:8080

Restart apache and kick off the scheduler:

sudo /etc/init.d/apache2 restart

/usr/local/autotest/scheduler/monitor_db.py /usr/local/autotest/results
Add your device to your setup (this requires you to be able to log into your machine as root).

src/third_party/autotest/files/cli/atest host create [ip of your host]
Setup a device with the image you want (important as the database will label your machine with its current image). Do this by kicking off the dummy suite.

src/third_party/autotest/files/site_utils/run_suite.py --board=[board] --build=[image you want to use] --suite=dummy
Now you should be able to kick off your server side telemetry tests through the AFE (go to localhost in your browser)! But in case your new test doesn't show up, run test importer. Then refresh the create job's page with cntl+shift+r

src/third_party/autotest/files/utils/test_importer.py

Note that the test will try to ssh to the dev_server, so make sure that 'ssh YOUR_IP_ADDRESS' works without prompting a password. If it doesn't, you'll need to setup public keys for your local machine. Following these steps:

First generate RSA ssh keys, see this link if you don't know how to. Assume you've created ~/.ssh/id_rsa,pub and ~/.ssh/id_rsa
Add the following to ~/.ssh/config

Host YOUR_IP_ADDRESS
User YOUR_USER_NAME
IdentityFile /path/to/home/.ssh/id_rsa
Add the content of ~/.ssh/id_rsa to ~/.ssh/authorized_keys

cat ~/.ssh/id_rsa >> ~/.ssh/authorized_keys
Restart ssh-agent

username@localhost~$ killall ssh-agent; eval `ssh-agent`
Allow ssh to your local machine using RSA keys

username@localhost~$ sudo vim /etc/ssh/sshd_config

# Turn on RSAAuthentication
RSAAuthentication yes
Restart ssh server

username@localhost~$ sudo service ssh restart
Test you can ssh to your local machine without password

username@localhost~$ ssh YOUR_IP_ADDRESS

Logging into Chrome from an Autotest (using Telemetry)

From an Autotest, you can use Telemetry to log into and out of Chrome using context management in Python (the "with/as" construct). To do so, your autotest must include the following import:

from autotest_lib.client.common_lib.cros import chrome

and then use the "with/as" construct like so:

with chrome.Chrome() as cr:

do_stuff() # Will be logged into Chrome here.

# Will be logged out at this point.

In this example, cr is a Telemetry Browser object. Wrapping your code inside of a "with/as" ensures that Chrome will be logged in at the start of the construct, and logged out at the end of the construct.

Rebooting a machine in a server-side test

This handles the simple reboot case:

def run_once(self, host=None):

host.reboot()

# If the reboot fails, it will raise error.AutoservRebootError

# If you need to recover from a reboot failure, note that calls

# to host.run() will almost certainly fail with a timeout.

If you have custom code that brings the DUT down, and then brings it back up again, you will need to use a slightly different flow:

def run_once(self, host=None):

boot_id = host.get_boot_id()

self._trigger_complicated_dut_restart()

host.wait_for_restart(old_boot_id=boot_id)

Running a client side test as part of a server side test

Inside a server job you can utilize the host object to create a client object to run tests.

def run_once(self, host, job_repo_url=None):

client_at = autotest.Autotest(host)

client_at.run_test('sleeptest')

*The status of this test is communicated through the status.log, if this test fails, the overall run of the job will be considered a failure.

Writing code to automatically generate a label for a DUT

Many tests have requirements for specific hardware characteristics, specified by setting DEPENDENCIES. So, imagine there's a new fubar class of device present on some new hardware. You have tests that exercise the driver for the new devices. Obviously, your test shouldn't run unless a fubar device is present. So, your control file will say this:

DEPENDENCIES = "fubar"

Alternatively, imagine that there are three kinds of fubar, types A, B, with the possibility of other types that might be added in future. There are boards with types A and B to be tested. You then have two control files, one for each type.

For Type A:

DEPENDENCIES = "fubar_type:type_a"

For Type B:

DEPENDENCIES = "fubar_type:type_b"

These hardware characteristics are meant be detected automatically when the DUT is added into the lab database. To make the detection happen, you have to write detection code. Here's how to do it.

The detection function must be added as a method in class CrosHost, found in file server/hosts/cros_host.py. The method must be decorated with @label_decorator. The return value is a string with the label name as it will appear in the DEPENDENCIES setting.

A sample of a simple binary label:

@label_decorator('fubar')

def has_fubar(self):

result = self.run('test -d /sys/class/fubar', ignore_status=True)

if result.exit_code != 0:

return None

return 'fubar'

A sample of a label with multiple values:

@label_decorator('fubar_type')

def get_fubar_type(self):

result = self.run('get_fubar_type', ignore_status=True)

typestring = result.stdout.strip()

if typestring == 'Type A':

return 'fubar_type:type_a'

if typestring == 'Type B':

return 'fubar_type:type_b'

# typestring 'Type C' is a proposed industry standard, but

# it hasn't been finalized. There may be other as yet

# unknown types in the future.

return None

One point about labels with multiple values bears emphasis: It may be that you will find that some valid values of a label aren't yet needed for any test. In that case, return None for the unused case. Do not return a label value that is not needed by any existing test. Generally, adding labels is easy, but removing labels is hard. So, don't create a label name until you know exactly where and how it will be used.

Use Chrome Driver in ChromeOS Autotest Test

About Chrome Driver

Chrome Driver is a standalone server which implements WebDriver's wire protocol for Chromium. Through Chrome Driver, you can easily interact with Chromium browser with the power of browser automation provided by WebDriver. A list of WebDriver calls you can make can be found in Selenium 2.0 Documentation. Some useful calls include:

get(url): Loads a web page in the current browser session.
execute_script(script, *args): Synchronously Executes JavaScript in the current window/frame.
find_element_by_name(name): Finds an element by name.

Chrome Driver Binary

All ChromeOS test images shall have Chrome Driver binary installed in /usr/local/chromedriver/. The binary is updated to the same version of Chrome in that test image. That is, you will always be using the latest build of Chrome Driver.

If your test expects to run against a “stable” build of Chrome Driver binary, you will need to write your own code in your test to download the desired binary and replace the binary in /usr/local/chromedriver/.

How to use Chrome Driver in an Autotest test

Writing a test that uses Chrome Driver to interact with Chrome is easy. There is a wrapper class for using Chrome Driver available in ChromeOS/Autotest. The wrapper class, as a context manager type, and handles the following tasks for you:

Logs in ChromeOS using Telemetry.
Starts Chrome Driver with Remote mode on the Device under Test (DUT) and connects to the remote debug port of the Chrome instance after log in.
Exposes a driver instance for you to make any Chrome Driver calls.
Shutdowns Chrome Driver process, and logs out of ChromeOS.

To write a test, you can follow the example of test desktopui_UrlFetchWithChromeDriver. All you need to do to get started are basically:

import the wrapper class

from autotest_lib.client.common_lib.cros import chromedriver

Create an instance of the Chrome Driver, and make calls.

with chromedriver.chromedriver() as chromedriver_instance:

driver = chromedriver_instance.driver

# Here you can make standard Chrome Driver calls through the driver instance.

# For example, browse a given url with |driver.get(url)|

Comments

The Chromium Projects

Quick links

Other sites