This is a collection of patterns to help aid test developers in finding useful parts of Autotest and common pathways when writing tests.
Adding retries to Flaky testsWe have a mechanism to retry tests that have flaky failures. A retries count can be specified, and a failing test will be re-attempted until it either passes or the number of failed retries reaches the retry count, in which case the test will be considered failed. To add retries to your test, you simply need to add a JOB_RETRIES = N attribute to the control file for the test you would like to retry, where N is an integer giving the maximum number of times the test should be retried. Note that a previous method of retrying flaky tests was to use the RETRIES attribute. While this attribute is no longer in use, one might find this in commits merged in or before Aug 2018. This bug tracks the migration from RETRIES to JOB_RETRIES. Wrapping a GTest test in AutotestWrapping a Browerstest in AutotestWrapping a Telemetry test in AutotestTelemetry is replacing Pyauto as the new performance testing framework for Chrome as well as the new key way to write tests that interact with the UI. Writing a test that uses telemetry is a 2 step process:
from autotest_lib.server.cros import telemetry_runner
def run_benchmark(machine):
host = hosts.create_host(machine) job.run_test("telemetry_benchmarks", host=host, benchmark="octane", page_sets=["octane.json"]) parrallel_simple(run_benchmark, machine) Locally Testing a Wrapped Telemetry Test:Because the telemetry runner code requires the full lab infrastructure, you need to set up a local autotest frontend and devserver in order to locally test your telemetry changes.
Logging into Chrome from an Autotest (using Telemetry)From an Autotest, you can use Telemetry to log into and out of Chrome using context management in Python (the "with/as" construct). To do so, your autotest must include the following import:
from autotest_lib.client.common_lib.cros import chrome and then use the "with/as" construct like so:
with chrome.Chrome() as cr: do_stuff() # Will be logged into Chrome here. # Will be logged out at this point. In this example, cr is a Telemetry Browser object. Wrapping your code inside of a "with/as" ensures that Chrome will be logged in at the start of the construct, and logged out at the end of the construct.
Rebooting a machine in a server-side testThis handles the simple reboot case:
def run_once(self, host=None): host.reboot() # If the reboot fails, it will raise error.AutoservRebootError # If you need to recover from a reboot failure, note that calls # to host.run() will almost certainly fail with a timeout. If you have custom code that brings the DUT down, and then brings it back up again, you will need to use a slightly different flow: def run_once(self, host=None): boot_id = host.get_boot_id() self._trigger_complicated_dut_restart() host.wait_for_restart(old_boot_id=boot_id) Running a client side test as part of a server side testInside a server job you can utilize the host object to create a client object to run tests.
def run_once(self, host, job_repo_url=None): client_at = autotest.Autotest(host) client_at.run_test('sleeptest') *The status of this test is communicated through the status.log, if this test fails, the overall run of the job will be considered a failure.
Writing code to automatically generate a label for a DUTMany tests have requirements for specific hardware characteristics, specified by setting DEPENDENCIES. So, imagine there's a new fubar class of device present on some new hardware. You have tests that exercise the driver for the new devices. Obviously, your test shouldn't run unless a fubar device is present. So, your control file will say this: DEPENDENCIES = "fubar" For Type A: DEPENDENCIES = "fubar_type:type_a" DEPENDENCIES = "fubar_type:type_b" These hardware characteristics are meant be detected automatically when the DUT is added into the lab database. To make the detection happen, you have to write detection code. Here's how to do it. The detection function must be added as a method in class CrosHost , found in file server/hosts/cros_host.py. The method must be decorated with @label_decorator . The return value is a string with the label name as it will appear in the DEPENDENCIES setting.A sample of a simple binary label: @label_decorator('fubar') def has_fubar(self): result = self.run('test -d /sys/class/fubar', ignore_status=True) if result.exit_code != 0: return None return 'fubar' @label_decorator('fubar_type') def get_fubar_type(self): result = self.run('get_fubar_type', ignore_status=True) typestring = result.stdout.strip() if typestring == 'Type A': return 'fubar_type:type_a' if typestring == 'Type B': return 'fubar_type:type_b' # typestring 'Type C' is a proposed industry standard, but # it hasn't been finalized. There may be other as yet # unknown types in the future. return None One point about labels with multiple values bears emphasis: It may be that you will find that some valid values of a label aren't yet needed for any test. In that case, return None for the unused case. Do not return a label value that is not needed by any existing test. Generally, adding labels is easy, but removing labels is hard. So, don't create a label name until you know exactly where and how it will be used.Use Chrome Driver in ChromeOS Autotest TestAbout Chrome DriverChrome Driver is a standalone server which implements WebDriver's wire protocol for Chromium. Through Chrome Driver, you can easily interact with Chromium browser with the power of browser automation provided by WebDriver. A list of WebDriver calls you can make can be found in Selenium 2.0 Documentation. Some useful calls include:
Chrome Driver BinaryAll ChromeOS test images shall have Chrome Driver binary installed in /usr/local/chromedriver/. The binary is updated to the same version of Chrome in that test image. That is, you will always be using the latest build of Chrome Driver. If your test expects to run against a “stable” build of Chrome Driver binary, you will need to write your own code in your test to download the desired binary and replace the binary in /usr/local/chromedriver/. How to use Chrome Driver in an Autotest testWriting a test that uses Chrome Driver to interact with Chrome is easy. There is a wrapper class for using Chrome Driver available in ChromeOS/Autotest. The wrapper class, as a context manager type, and handles the following tasks for you:
To write a test, you can follow the example of test desktopui_UrlFetchWithChromeDriver. All you need to do to get started are basically:
from autotest_lib . client . common_lib . cros import chromedriver
|
Chromium OS > Testing Home >