Chromium OS‎ > ‎Testing Home‎ > ‎

Running Smoke Suite On a VM Image


Contents

  1. 1 Introduction
  2. 2 Smoke Suite tests
  3. 3 Running the Smoke Test Suite on a Virtual Machine (VM)
    1. 3.1 Useful Links:
    2. 3.2 Install qemu-kvm
    3. 3.3 Build a testable VM image
    4. 3.4 Start a VM image
    5. 3.5 Run the autotest tests on the VM image
    6. 3.6 Run the autotest tests on the VM image using cros_run_vm_test
    7. 3.7 Run parallel tests on mulitple VMs simultaneously
    8. 3.8 Troubleshooting
      1. 3.8.1 No space left on device
      2. 3.8.2 Could not set up host forwarding rule 'tcp::9222-:22'
      3. 3.8.3 Could not initialize KVM, will disable KVM support
  4. 4 Updating a client autotest
    1. 4.1 The new way: Use cros_workon with all Python tests
    2. 4.2 The old way: emerge individual autotest tests
    3. 4.3 Debugging

Introduction

The VMTest "smoke suite" is a suite of client-side autotests that performs basic sanity checks on a Chromium OS build.
Each of the tests in the smoke test suite is capable of running in a virtual machine.
This allows the smoke suite to be run as part of the automated build process on a 'buildbot'.

Smoke Suite tests

The Smoke suite currently consists of the following tests (this list is subject to change):
  • build_RootFilesystemSize
  • desktopui_ChromeFirstRender
  • desktopui_FlashSanityCheck
  • desktopui_KillRestart('^chrome$')
  • desktopui_KillRestart('^session_manager$')
  • logging_CrashSender
  • login_Backdoor('$backdoor')
  • login_BadAuthentication
  • login_CryptohomeIncognitoMounted('$backdoor')
  • login_CryptohomeMounted
  • login_LoginSuccess
  • platform_FilePerms
  • platform_OSLimits

Running the Smoke Test Suite on a Virtual Machine (VM)

Useful Links:

The instructions on this page should be enough to get you going.  They install a testable Chromium OS image on a VM and run the smoke suite.  However, if you want more details, you can always go to one of the following pages:

Install qemu-kvm

First of all, to run in a vm, you must have vm software installed.
On Ubuntu Lucid, use apt-get to install qemu-kvm:

$ sudo apt-get install qemu-kvm

Build a testable VM image

WARNING: After crbug/710629, 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so is the most likely to work at ToT. 'betty' is based on 'amd64-generic', though, so 'amd64-generic' is likely to also work for most (non-ARC) tests.

Next, build the testable VM image.  The following steps assume that:
  1. depot_tools/repo have been installed and configured.
  2. Chromium OS source code has been checked out.
  3. A chroot has been created with make_chroot.

$ cros_sdk    # To enter the chroot

# Setup the build environment for the target board.  ${BOARD} could should usually be 'betty' or 'amd64-generic'.
(chroot)$ ./setup_board --board=${BOARD}

# Build all the source packages, including those required for running autotests
# Optional: "--oldchromebinary" may make your build faster, but will have old Chrome/libcros
# NOTE: At the moment, "--withautotest" is the default, so don't worry if you built without it.
(chroot)$ ./build_packages --board=${BOARD}

# Build a bootable image
# Optional: Include "--noenable_rootfs_verification" if you think you might need to modify your rootfs.
(chroot)$ ./build_image --board=${BOARD} test

# Clone this image, modify it for test, and make image for use in qemu-kvm Virtual Machine
# Note: because we use "--test_image", an explicit "modify_image_for_test" is not required.
(chroot)$ ./image_to_vm.sh --board=${BOARD} --test_image

# Exit chroot
(chroot)$ exit

The newly created VM image should be in the following path (relative to src/scripts):
../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

Start a VM image

Before launching autotests, its fun to boot up a virtual Chromium OS on your desktop!

$ ./bin/cros_start_vm --image_path=../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

However, running autotests is much slower when the VM must render graphics (especially without hardware acceleration).  
So, we typically start kvm in '-nographic' mode, like this:

$ ./bin/cros_start_vm --no_graphics --image_path=../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

# ssh into port 9222 to access the VM
$ ssh root@localhost -p 9222 -o StrictHostKeyChecking=no
Last login: ... from ... on ...
localhost ~ # exit
logout
Connection to localhost closed.

Note that test images have the root password changed to test0000.

Run the autotest tests on the VM image

Autotests can be run against the VM via test_that.

# Run the entire smoke suite
(chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke

# Run an individual test from the smoke Suite
(chroot)$ test_that --board=${BOARD} localhost:9222 logging_CrashSender

Don't forget to kill the head-less VM when you are done with it!

$ ssh root@localhost -p 9222 -o StrictHostKeyChecking=no
Last login: ... from ... on ...
localhost ~ # shutdown -H 0
localhost ~ # exit
logout
Connection to localhost closed.

Run the autotest tests on the VM image using cros_run_vm_test

Another script, bin/cros_run_vm_test, can be used to start the VM, run autotest tests on it, and then shut down the VM, all from a single invocation:

# Run the entire smoke suite:
$ ./bin/cros_run_vm_test --no_graphics --board=${BOARD} suite:smoke

# Run an individual test from the suite
$ ./bin/cros_run_vm_test --no_graphics --board=${BOARD} platform_OSLimits

Run parallel tests on mulitple VMs simultaneously

There is an ton of overhead involved in running a single autotest test.  To help speed things up, it is possible to run multiple tests in parallel, by spawning multiple VMs, each of which is listening on a different SSH port using bin/cros_run_parallel_vm_test.py.

# Run the smoke test suite... in parallel!
$ ./bin/cros_run_parallel_vm_tests.py --board=${BOARD} build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits

Troubleshooting

No space left on device

If you get an error that looks like this:

Unhandled OSError: [Errno 28] No space left on device: '/home/autotest/tmp/_autotmp_Woz_Qyharness-fifo'

...this may be because you have run out of space on your "stateful partition".  This can happen if you leave the VM running for a long time and errors fill up the /var/log folder.  In my case, this folder contained 250M of data!  You should go into /var/log and cleanup stuff.  

Use df to determine how much space is available in the stateful partition:

localhost ~ # df /mnt/stateful_partition
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              2064208    450612   1508740  23% /mnt/stateful_partition

You may be getting lots of errors if you're running a board-specific image with drivers for hardware that is not present on the VM.  If this is the case, try stopping the responsible daemon with stop <daemon_name>.

Could not set up host forwarding rule 'tcp::9222-:22'

If you get this error when running cros_start_vm, make sure that you don't have a VM already running. This should kill it for you.

sudo pkill -f qemu

Could not initialize KVM, will disable KVM support

This is probably because you don't have the kvm module loaded. The VM will run.....very.......slowly.

open /dev/kvm: No such file or directory
Could not initialize KVM, will disable KVM support
$ modprobe kvm_intel    # if you have an AMD machine, use 'kvm_amd'

Note that this module interferes with VirtualBox so you may need to remove it (rmmod kvm_intel) if you want to use this.

Updating a client autotest

Autotest tests are usually built and compiled on the autotest server machine.
A 'client' test is installed on the client by the autotest infrastructure when the test is run.
The autotest infrastructure is maintained in the chromeos-base/autotest cros_workon ebuild.
Individual tests and test suites are in the chromeos-base/autotest-tests ebuild.

The new way: Use cros_workon with all Python tests

Working on all Python autotests should be simple and fast.
First cros_workon autotest for the board on which you plan to run tests:

$ cros_workon start autotest --board=${BOARD}

Now, test_that will detect that you are working on autotests and automatically copy over Python in the autotest source tree to the sysroot.

Run test_that as usual:

(chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke

The old way: emerge individual autotest tests

If you have a test with binary dependencies, you must rebuild them explicitly:

(chroot)$ emerge-${BUILD} autotest-tests

This can take a very, very long time since it will rebuild ALL tests.
To speed this up, set the TESTS variable on the command line when runnning emerge to install an individual test:

(chroot)$ TESTS=<TestName> emerge-${BOARD} autotest-tests

Note that doing this will deactivate all other tests! This is a problem for test suites.
To rebuild a whole test suite, specify the suite and all individual tests in the suite.  For example, to rebuild smoke suite:

(chroot)$ TESTS='build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits' \
emerge-${BOARD} autotest-tests

Then go grab a coffee...

Debugging

You can use chromium os debugging tps on vm as well.

Comments