Chromium OS‎ > ‎Testing Home‎ > ‎MobLab‎ > ‎

Using MobLab

***DEPRECATED, PLEASE VISIT https://sites.google.com/a/chromium.org/dev/chromium-os/testing/moblab *** (20170714)


This section describes how to perform basic MobLab tasks. Full, detailed AFE documentation is available here.
NOTE: Using MobLab requires internet connection, to access Google Cloud Storage for getting the right software images and for storing test results.

Accessing the Moblab Autotest Frontend (AFE)

  1. To access the AFE from the MobLab itself, go to http://localhost/ in a web browser.
  2. To access the AFE from any networked machine, go to http://<moblab_server_IPaddress> in a web browser, where moblab_server_IPaddress is the external IP address of the MobLab server (assigned by your ISP).
  3. SNAX directions for internal users at Google:  You may not be able to access the AFE from another machine using the directions above unless your machine is on the hub.

Register hosts

To enable viewing hosts and running tests on them in the AFE, you register each host to the application. To enable running test suites on a group of machines, you also assign a pool: label.
  1. Check the IP address of each host (DUT) by running ifconfig at a bash command prompt on the host.  The IP address is assigned from a IP address pool of 192.168.231.100-199.
  2. Open the AFE from a remote machine in a browser window.
  3. Click the Admin Interface link at the top right of the page.
  4. In the Hosts row, click Add.
  5. Type the IP address of a host on the test subnet in the Hostname field.  Make sure that “Labels autodetection” is checked”.
  6. Click Save.
To enable running test suites:
  1. go to Admin Interface to Create a new label(click the "Add" link next to "Labels") pool:<pool_name> where pool_name is a text string describing the set of hosts. For example, pool:wifi6 in "Name" fieldof Add Label dialog. 
  2. Specify the label in the Label field for each host in the Add Host Dialog shown in 4. above.
The pool:<pool_name> label is required for the set of hosts on the subnet that need to run a test suite. 

Running Tests from the AFE

With Autotest, tests can run according to a schedule (recurring), or manually. Recurring tests are configured once and always use the latest Chrome OS Tip-of-Tree (ToT) test image. Typically, you create and run a test manually before running it on a schedule.

(Advanced User) Additional information on Autotest: For descriptions of each parameter in the Create Job view or details of how the Autotest scheduler runs jobs, click here.

Running your first test from the AFE

  1. Click the tab to open the Create Job view. 

  1. Type a name for the job. The string should be meaningful when viewing a list of jobs.
  1. Type the name of the test image to use in the Image URL/Build field.  If no image is specified and you are not running a suite job, the latest ToT image is used. For some  tests Image URL/Build is a required field. In such cases add the release name here in the following format: <board>-release/RXX-YYYY.ZZ.0 eg. samus-release/R42-6812.88.0. 

  1. Select the tests to run. A good visual first test to try is graphics_WebGLManyPlanetsDeep

  1. Click the names of registered hosts for running the tests.

  1. Click Submit Job.

Run a suite of tests from the AFE

  1. Click the tab to open the Create Job view.
  2. Type a name for the job. The string should be meaningful when viewing a list of jobs.
  3. (Optional) Type the name of the test image to use in the Image URL/Build field . For example, mario-release/R45-7141.0.0
  4. Select Server from the Test type dropdown list.
  1. Select test suites to run.

 
  1. If you select a suite, the hostless (suite job) attribute will automatically be selected in the “Advanced Options” section.
  1. (Optional) If you want to run a suit of tests on a pool of hosts you have defined during "Resiter hosts", specify the pool_name in the Pool: (optional) field.
  2. Click Submit Job.

Setting up recurring tests

  1. Configure the job in the Create Job view and click Submit Job.
  2. Click the job in the Job List view.
  3. Click the Create recurring job button.
  4. Specify a start time, a delay between each run, and the number of times to run the test.
  5. Click Create recurring job.

Viewing running jobs using the AFE

When you access the AFE, the default view is Job List. It provides a list of jobs on the MobLab server owned by the current user. Other available views are tabs in the horizontal navigation bar.
  • To see jobs owned by all users, click the All users option for Owner.
  • To see jobs with a specific status, select Queued Jobs, Running Jobs, or Finished Jobs below the horizontal navigation bar.
The following table lists views and gives a description of each:
View Name Description
Job List List of all jobs on the MobLab server, including job ID, name, status.
 View Job  Detailed view of the selected job.
 Recurring Job Form for scheduling and cancelling recurring tests. 
Create Job   Form for configuring and running manual tests.
Host List  List of all host currently on the test subnet. 
View Host  List of all test jobs run on the selected host. 
 User Preferences  

Viewing results from the AFE

Click the line for the job in the Job List view.  The detail record contains the results of the test run.

To view a full test report, with details and a test matrix, click the Open in new window link.



Staging Custom Images & Client-side Tests on MobLab

Overview

MobLabs allow Chrome OS partners to do the same testing Google does in-house at their locations. Previously MobLab only supported testing on Chrome OS images built by Google.

Now with a new cros stage tool, MobLab users can test their own locally built (or other custom images), eliminating the need for changes to be submitted to the main ChromeOS tree prior to testing. 

Users will be able to push custom images directly to a local MobLab or to Google Storage for remote MobLabs to access.

A key feature of cros stage is that client side tests will always from the chroot from where cros stage is being called.  This allows users to run locally written tests on both locally-built builds as well as official CrOS builds stored in their Google Cloud bucket.

Requirements

  • A MobLab running version R42-6715.0.0 or newer. Your MobLab should autoupdate to this version or a newer one automatically.
    • To check your MobLab version:
      • Log in via the UI and navigate to ‘chrome://chrome’ in the browser.
      • SSH in and run `cat /etc/lsb-release | grep CHROMEOS_RELEASE_DESCRIPTION’
  • An up-to-date chroot that has access to the ‘cros stage’ tool. Repo sync’ing the whole tree or the chromite/ directory to tip of tree will provide the tool.
  • For uploading to Google Storage, a private Google Storage Bucket and a corresponding boto key are required. Partners should have received this from their Google Partner Engineering Representative.

Usage

Building a custom test image

In order to test a locally built test image you must first build one in the chroot. Instructions for this can be found in the Chromium OS developer guide.

Notes

  • Only test images are supported, using a Non-Test image will make your DUT unable to be tested in the future and will require manual reformatting.
  • The autotest bits used to test this image will be the latest in your build sysroot! I.E. if you emerge new autotest changes after producing the image you wish to stage, these changes will not match.

Staging a custom image on a local MobLab

To stage on a local MobLab that is accessible by your chroot you simply use the ‘cros stage’ command:

cros stage </path/to/board/build/chromiumos-test-image.bin> <MobLab IP address>

Example:

In this example I am pushing an image I built in my chroot to the MobLab device.

~/trunk/src/scripts $ cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin 100.XX.XX.XX

The path requires the format of /board/build/chromiumos-test-image.bin.

Once completed ‘cros stage’ will output the name of the image to use when kicking off tests via the run_suite command or the AFE. For this example: peppy-custom/R41-6576.0.2014_12_12_1535

Notes:

  • The custom image will only stay on the Moblab device for 24 hours at which point it will be automatically wiped.

Staging an official image with custom tests on a local MobLab

You can also stage official images from Google Storage with test packages from your chroot onto a local Moblab.  The command, cros stage always pulls test packages from the chroot regardless of the image source.

For this work flow, you must have previously built an image for this board at some point. For information about doing custom test development please refer to the Advanced:  Custom Client-Side Tests Support.

cros stage gs://<bucket-name>/<board>-<builder type>/<build name>/ <MobLab IP address>

Example:

In this example I am pushing an official image with test packages built in my chroot to the MobLab device.  I would have already built a squawks test image at some point.

~/trunk/src/scripts $ cros stage gs://chromeos-image-archive/squawks-release/R42-6750.0.0 100.XX.XX.XX

Once completed cros stage will output the name of the image to use when kicking off tests. For this example: squawks-custom/R42-6570.0.0

Staging a custom image to Google Storage

You can also push images to Google Storage. The advantages to this is that the image will not be deleted for 24 hours, and that remote MobLab’s not in the same local network can access the image as well.

cros stage  </path/to/board/build/chromiumos-test-image.bin> <gs_base_path>    --boto_file=<boto_file_path>

Example:

cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin gs://chromeos-moblab-peppy --boto_file=~/peppy.boto

Once completed ‘cros stage’ will output the name of the image to use when kicking off tests just like when staging on a local MobLab.

Testing the custom image

Simply run a test like you would do before but specify the outputted image name when launching the test:


Advanced: Custom Client-Side Tests Support

NOTE: Custom Client-Side Tests Workflow is only supported when using cros stage with a Moblab address and NOT when using an Google Storage URL as the staging destination.

Cros Stage also enables developer to run Custom Client Side Tests on their MobLab. It does this by packaging up the tests that are built and stored in the board’s buildroot (/build/<board>/) of the chroot.

  1. First make your test changes in the autotest repo (src/third_party/autotest/files).
  2. If this is a new test, add it to an ebuild to ensure it is built at build time.
    • Look at src/third_party/chromiumos-overlays/chromeos-base/autotest-tests* ebuilds for examples.
  3. If this is an existing test, note the ebuild it belongs to.
    • grep -r *<test_name>* src/third_party/chromiumos-overlays/chromeos-base
  4. Build your test changes. From a chroot:
    • cros_workon start --board=<board> <ebuild-name>
    • emerge-<board> <ebuild-name>
  5. Run cros stage from the chroot to stage your image and test packages.
    • cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin 100.XX.XX.XX
Note you do not need to build a new image. Therefore you can use the same image repeatedly while doing test development.

For more information:

Advanced Tasks

Using command line tools to run tests

You can also use the Autotest command line interface (CLI) to run manual tests. The following Python utilities are available to MobLab users for running tests:
  • atest
  • run_suite
For a more extensive list, click here.

To run a test or test suite from the command line, use the following syntax:
run_suite.py --board=<board_name> --build=<build_name> --suite_name=<suite_name> --pool=””

For example:
run_suite.py --board=peppy --build=peppy-release/R37.5847.0.0 --suite_name=smoke --pool=””

To obtain the names of test builds, use gsutil to browse your Google Cloud Storage bucket.

You can check job status and results through the AFE. This page describes the status values for hosts and jobs.

Writing Custom Tests

The MobLab server runs the open source git Autotest framework. The framework supports writing custom tests in Python using the Autotest APIs.

For details and examples of coding to the Autotest APIs, view the general user documentation.  Please note that although MobLab uses the Autotest framework, code refactoring has made it so that many examples do not work with our code base. However, many of these documents are still helpful at a high level.  

Chromium OS-specific information can be found here and our repo is located here.

Booting from a Recovery Image

If the MobLab server hangs and the device can’t be manually rebooted, it is possible to boot from a recovery image. For Chrome OS partners who have deployed MobLab, the recovery image is located in a folder in Google Cloud Storage. It is not generally available on the Chrome OS image release site.  

To boot MobLab from a recovery image:
  1. Copy the MobLab recovery image to a USB memory stick using cros flash:  cros flash usb:// remote/stumpy_moblab/latest-dev/recovery
  2. This example copies the latest recovery image in the dev channel to a USB drive.
  3. From the MobLab device (in Developer Mode), press Ctrl-Alt-F2 to switch to the second virtual terminal.
  4. Run crossystem dev_boot_usb=1.
  5. Reboot the MobLab device.
  6. Insert the USB memory stick with the recovery image into an open USB port on the MobLab device.
  7. When Chrome OS displays the Developer Mode screen, press Ctrl-U to boot from a USB device

Advanced Commands

These commands have been added to the MobLab user profile so you can run them anywhere on the system. Please note that this list is valid as of version 6343.

  • run_suite.py - Command line tool for creating suite jobs.
    • Example: run_suite.py --board=peppy --build=peppy-release/R37.5847.0.0 --suite_name=smoke --pool=””
  • atest - Autotest command line tool able to add and edit hosts, labels and jobs in the database directly. For more information use --help option. Help is available for all the possible subcommands.
  • host_add - Add a test device. Shortcut for atest host create
    • example: host_add 192.168.231.100
  • host_delete - Delete a test device. Shortcut for atest host delete
    • exam</li></ul>ple: host_delete 192.168.231.100
  • label_create - Create a new label. Shortcut for atest label create
    • example: label_create board:peppy
  • label_delete - Delete a label. Shortcut for atest label delete
    • example: label_delete board:peppy
  • label_add - Add a label to a machine. Shortcut for atest label add
    • example: label_add board:peppy -m 192.168.231.100
  • abort_jobs - Abort queued or running jobs. Shortcut for atest job abort
    • example: abort_jobs 50 51 52
  • edit_moblab_config - Open vi to edit moblab’s shadow_config.
  • import_tests - Import any new tests located in /usr/local/autotest/{client,server}/site_tests so that they can be run from the AFE. Note: These custom tests in these directories will be overwritten if MobLab is powerwashed or if a new test with the same name is checked into the source.

Information on how MobLab works

Bringing up the test environment

After MobLab is connected to a network and booted, the MobLab server performs the following network configuration tasks:
  1. Connects to the test subnet through a USB Ethernet adapter.
  2. Requests subnet addresses 192.168.231.x from the DHCP server.
  3. Configures a static IP address for the MobLab server of 192.168.231.1 through the USB Ethernet adapter.
  4. Connects to the corporate network through the on-device Ethernet port.
  5. Obtains an IP address from the DHCP server on the corporate network. This is the external IP address of the MobLab server.
  6. Configures a gateway service for host Internet access.
  7. Enables IP forwarding.
  8. Adds a NAT module to the Chrome OS kernel.
  9. Enables automatic updates of Chrome OS and the test suite.
  10. Uses SSH to configure the server settings.
One or more hosts are connected to the test subnet by either a switch or a hub. MobLab communicates with the test subnet using the USB Ethernet adapter. When the MobLab server boots up, it waits up to 5 minutes to establish an SSH connection through the USB adapter and subnet. Then it assigns dynamic IP addresses to all hosts, ranging from 192.168.231.100 to 192.168.231.199.

Access to the partner’s corporate network enables automatic software updates for the MobLab server. It also enables access to Google Cloud Storage for downloading test images and server updates, and uploading logs. The corporate network should provide security features (firewall, VPN, etc.) to protect MobLab and the test subnet from potential attacks from outside the lab.

Automatic updates

The Autotest scheduler manages and coordinates build updates. On each MobLab device, it manages two separate types of build images:
  • The MobLab server image
  • Test images, for pushing to hosts
The scheduler waits for a time of low or no activity on the MobLab server to update the test image. If tests are running when an image is updated, they continue to run.

Automatic disk space management

After tests run, the Autotest logs are archived in your private Google Cloud Storage bucket. MobLab runs a log cleanup script as a background process and every 12 hours the script removes log files, making space for more test results.

By default, test logs are visible to the Chrome OS team at Google. If specific tests contain confidential data, you can control access to those test logs.
Comments