Chromium OS‎ > ‎Testing Home‎ > ‎

Running Smoke Suite On a VM Image


The VMTest "smoke suite" is a suite of client-side autotests that performs basic sanity checks on a Chromium OS build.
Each of the tests in the smoke test suite is capable of running in a virtual machine.
This allows the smoke suite to be run as part of the automated build process on a 'buildbot'.

Smoke Suite tests

The Smoke suite currently consists of the following tests (this list is subject to change):
  • build_RootFilesystemSize
  • desktopui_ChromeFirstRender
  • desktopui_FlashSanityCheck
  • desktopui_KillRestart('^chrome$')
  • desktopui_KillRestart('^session_manager$')
  • logging_CrashSender
  • login_Backdoor('$backdoor')
  • login_BadAuthentication
  • login_CryptohomeIncognitoMounted('$backdoor')
  • login_CryptohomeMounted
  • login_LoginSuccess
  • platform_FilePerms
  • platform_OSLimits

Running the Smoke Test Suite on a Virtual Machine (VM)

Useful Links:

The instructions on this page should be enough to get you going.  They install a testable Chromium OS image on a VM and run the smoke suite.  However, if you want more details, you can always go to one of the following pages:

Install qemu-kvm

First of all, to run in a vm, you must have vm software installed.
On Ubuntu Lucid, use apt-get to install qemu-kvm:

$ sudo apt-get install qemu-kvm

Build a testable VM image

Next, build the testable VM image.  The following steps assume that:
  1. depot_tools/repo have been installed and configured.
  2. Chromium OS source code has been checked out.
  3. A chroot has been created with make_chroot.

$ cros_sdk    # To enter the chroot

# Setup the build environment for the target board.  ${BOARD} could be a specific target platform, or x86-generic.
(chroot)$ ./setup_board --board=${BOARD}

# Build all the source packages, including those required for running autotests
# Optional: "--oldchromebinary" may make your build faster, but will have old Chrome/libcros
# NOTE: At the moment, "--withautotest" is the default, so don't worry if you built without it.
(chroot)$ ./build_packages --board=${BOARD}

# Build a bootable image
# Optional: Include "--noenable_rootfs_verification" if you think you might need to modify your rootfs.
(chroot)$ ./build_image --board=${BOARD} test

# Clone this image, modify it for test, and make image for use in qemu-kvm Virtual Machine
# Note: because we use "--test_image", an explicit "modify_image_for_test" is not required.
(chroot)$ ./ --board=${BOARD} --test_image

# Exit chroot
(chroot)$ exit

The newly created VM image should be in the following path (relative to src/scripts):

Start a VM image

Before launching autotests, its fun to boot up a virtual Chromium OS on your desktop!

$ ./bin/cros_start_vm --image_path=../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

However, running autotests is much slower when the VM must render graphics (especially without hardware acceleration).  
So, we typically start kvm in '-nographic' mode, like this:

$ ./bin/cros_start_vm --no_graphics --image_path=../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

# ssh into port 9222 to access the VM
$ ssh root@localhost -p 9222 -o StrictHostKeyChecking=no
Last login: ... from ... on ...
localhost ~ # exit
Connection to localhost closed.

Note that test images have the root password changed to test0000.

Run the autotest tests on the VM image

Usually autotests are invoked using

# Run the entire smoke suite
(chroot)$ ./ --remote=localhost --ssh_port 9222 --board=${BOARD} --use_emerged suite:smoke

# Run an individual test from the smoke Suite
(chroot)$ ./ --remote=localhost --ssh_port 9222 --board=${BOARD} --use_emerged logging_CrashSender

Don't forget to kill the head-less VM when you are done with it!

$ ssh root@localhost -p 9222 -o StrictHostKeyChecking=no
Last login: ... from ... on ...
localhost ~ # shutdown -H 0
localhost ~ # exit
Connection to localhost closed.

Run the autotest tests on the VM image using cros_run_vm_test

Another script, bin/cros_run_vm_test, can be used to start the VM, run autotest tests on it, and then shut down the VM, all from a single invocation:

# Run the entire smoke suite:
$ ./bin/cros_run_vm_test --no_graphics --board=${BOARD} suite:smoke

# Run an individual test from the suite
$ ./bin/cros_run_vm_test --no_graphics --board=${BOARD} platform_OSLimits

Run parallel tests on mulitple VMs simultaneously

There is an ton of overhead involved in running a single autotest test.  To help speed things up, it is possible to run multiple tests in parallel, by spawning multiple VMs, each of which is listening on a different SSH port using bin/

# Run the smoke test suite... in parallel!
./bin/ --board=${BOARD} build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits


No space left on device

If you get an error that looks like this:

Unhandled OSError: [Errno 28] No space left on device: '/home/autotest/tmp/_autotmp_Woz_Qyharness-fifo'

...this may be because you have run out of space on your "stateful partition".  This can happen if you leave the VM running for a long time and errors fill up the /var/log folder.  In my case, this folder contained 250M of data!  You should go into /var/log and cleanup stuff.  

Use df to determine how much space is available in the stateful partition:

localhost ~ # df /mnt/stateful_partition
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              2064208    450612   1508740  23% /mnt/stateful_partition

You may be getting lots of errors if you're running a board-specific image with drivers for hardware that is not present on the VM.  If this is the case, try stopping the responsible daemon with stop <daemon_name>.

Could not set up host forwarding rule 'tcp::9222-:22'

If you get this error when running cros_start_vm, make sure that you don't have a VM already running. This should kill it for you.

sudo pkill -f qemu

Could not initialize KVM, will disable KVM support

This is probably because you don't have the kvm module loaded. The VM will run.....very.......slowly.

open /dev/kvm: No such file or directory
Could not initialize KVM, will disable KVM support
$ modprobe kvm_intel    # if you have an AMD machine, use 'kvm_amd'

Note that this module interferes with VirtualBox so you may need to remove it (rmmod kvm_intel) if you want to use this.

Updating a client autotest

Autotest tests are usually built and compiled on the autotest server machine.
A 'client' test is installed on the client by the autotest infrastructure when the test is run.
The autotest infrastructure is maintained in the chromeos-base/autotest cros_workon ebuild.
Individual tests and test suites are in the chromeos-base/autotest-tests ebuild.

The new way: Use cros_workon

Working on autotests should be simple and fast.
First cros_workon autotest-tests or autotest for the board on which you plan to run tests:

$ cros_workon start autotest-tests --board=${BOARD}

Now, passing --build to will detect that you are working on autotests and automatically rebuild the requested tests and their dependencies using local sources.

(chroot)$ ./ --board=${BOARD} --build --remote=localhost --ssh_port 9222 --use_emerged suite:smoke

It is still possible to run tests with previously emerged autotest and autotest-tests packages, by passing --use_emerged flag to

The old way: emerge individual autotest tests

In the past, if you made a change to an individual client test, you had to re-emerge the auto-test package:

(chroot)$ emerge-${BUILD} autotest-tests

This can take a very, very long time since it will rebuild ALL tests.
To speed this up, set the TESTS variable on the command line when runnning emerge to install an individual test:

(chroot)$ TESTS=<TestName> emerge-${BOARD} autotest-tests

Note that doing this will deactivate all other tests! This is a problem for test suites.
To rebuild a whole test suite, specify the suite and all individual tests in the suite.  For example, to rebuild smoke suite:

(chroot)$ TESTS='build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits' \
emerge-${BOARD} autotest-tests

Then go grab a coffee...


You can use chhromium os debugging tps on vm as well.