Chromium OS‎ > ‎Testing Home‎ > ‎

Running Smoke Suite On a VM Image


Contents

  1. 1 Introduction
  2. 2 Smoke Suite tests
  3. 3 Running the Smoke Test Suite on a Virtual Machine (VM)
    1. 3.1 Useful Links:
    2. 3.2 Install qemu-kvm
    3. 3.3 Build a testable VM image
    4. 3.4 Start a VM image
    5. 3.5 Run the autotest tests on the VM image
    6. 3.6 Run the autotest tests on the VM image using cros_run_vm_test
    7. 3.7 Troubleshooting
      1. 3.7.1 No space left on device
      2. 3.7.2 Could not set up host forwarding rule 'tcp::9222-:22'
      3. 3.7.3 Could not initialize KVM, will disable KVM support
  4. 4 Updating a client autotest
    1. 4.1 The new way: Use cros_workon with all Python tests
    2. 4.2 The old way: emerge individual autotest tests
    3. 4.3 Debugging

Introduction

The VMTest "smoke suite" is a suite of client-side autotests that performs basic sanity checks on a Chromium OS build.
Each of the tests in the smoke test suite is capable of running in a virtual machine.
This allows the smoke suite to be run as part of the automated build process on a 'builder'.

Smoke Suite tests

The Smoke suite currently consists of the following tests (this list is subject to change):
  • build_RootFilesystemSize
  • desktopui_ChromeFirstRender
  • desktopui_FlashSanityCheck
  • desktopui_KillRestart('^chrome$')
  • desktopui_KillRestart('^session_manager$')
  • logging_CrashSender
  • login_Backdoor('$backdoor')
  • login_BadAuthentication
  • login_CryptohomeIncognitoMounted('$backdoor')
  • login_CryptohomeMounted
  • login_LoginSuccess
  • platform_FilePerms
  • platform_OSLimits

Running the Smoke Test Suite on a Virtual Machine (VM)

Useful Links:

The instructions on this page should be enough to get you going.  They install a testable Chromium OS image on a VM and run the smoke suite.  However, if you want more details, you can always go to one of the following pages:

Install qemu-kvm

First of all, to run in a vm, you must have vm software installed.
On Ubuntu Lucid, use apt-get to install qemu-kvm:

$ sudo apt-get install qemu-kvm

Build a testable VM image

WARNING: After crbug/710629, 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so is the most likely to work at ToT. 'betty' is based on 'amd64-generic', though, so 'amd64-generic' is likely to also work for most (non-ARC) tests.

Next, build the testable VM image.  The following steps assume that:
  1. depot_tools/repo have been installed and configured.
  2. Chromium OS source code has been checked out.
  3. A chroot has been created with make_chroot.

$ cros_sdk    # To enter the chroot

# Setup the build environment for the target board.  ${BOARD} could should usually be 'betty' or 'amd64-generic'.
(chroot)$ ./setup_board --board=${BOARD}

# Build all the source packages, including those required for running autotests
# Optional: "--oldchromebinary" may make your build faster, but will have old Chrome/libcros
# NOTE: At the moment, "--withautotest" is the default, so don't worry if you built without it.
(chroot)$ ./build_packages --board=${BOARD}

# Build a bootable image
# Optional: Include "--noenable_rootfs_verification" if you think you might need to modify your rootfs.
(chroot)$ ./build_image --board=${BOARD} test

# Clone this image, modify it for test, and make image for use in qemu-kvm Virtual Machine
# Note: because we use "--test_image", an explicit "modify_image_for_test" is not required.
(chroot)$ ./image_to_vm.sh --board=${BOARD} --test_image

# Exit chroot
(chroot)$ exit

The newly created VM image should be in the following path (relative to src/scripts):
../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

Start a VM image

Before launching autotests, its fun to boot up a virtual Chromium OS on your desktop!

(chroot)$ cros_vm --start --board=${BOARD}

Run the autotest tests on the VM image

Autotests can be run against the VM via test_that.

# Run the entire smoke suite
(chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke

# Run an individual test from the smoke Suite
(chroot)$ test_that --board=${BOARD} localhost:9222 logging_CrashSender

Don't forget to kill the head-less VM when you are done with it!

(chroot)$ cros_vm --stop

Run the autotest tests on the VM image using cros_run_vm_test

Another script, cros_run_vm_test, can be used to start the VM, run autotest tests on it, and then shut down the VM, all from a single invocation:

# Run the entire smoke suite:
(chroot)$ mkdir results
(chroot)$ cros_run_vm_test --board=${BOARD} --results-dir=results --autotest suite:smoke

# Run an individual test from the suite
(chroot)$ cros_run_vm_test --board=${BOARD} --results-dir=results --autotest platform_OSLimits

Troubleshooting

No space left on device

If you get an error that looks like this:

Unhandled OSError: [Errno 28] No space left on device: '/home/autotest/tmp/_autotmp_Woz_Qyharness-fifo'

...this may be because you have run out of space on your "stateful partition".  This can happen if you leave the VM running for a long time and errors fill up the /var/log folder.  In my case, this folder contained 250M of data!  You should go into /var/log and cleanup stuff.  

Use df to determine how much space is available in the stateful partition:

localhost ~ # df /mnt/stateful_partition
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              2064208    450612   1508740  23% /mnt/stateful_partition

You may be getting lots of errors if you're running a board-specific image with drivers for hardware that is not present on the VM.  If this is the case, try stopping the responsible daemon with stop <daemon_name>.

Could not set up host forwarding rule 'tcp::9222-:22'

If you get this error when running cros_vm --start, make sure that you don't have a VM already running. This should kill it for you.

cros_vm --stop

Could not initialize KVM, will disable KVM support

This is probably because you don't have the kvm module loaded. The VM will run.....very.......slowly.

open /dev/kvm: No such file or directory
Could not initialize KVM, will disable KVM support
$ modprobe kvm_intel    # if you have an AMD machine, use 'kvm_amd'

Note that this module interferes with VirtualBox so you may need to remove it (rmmod kvm_intel) if you want to use this.

Updating a client autotest

Autotest tests are usually built and compiled on the autotest server machine.
A 'client' test is installed on the client by the autotest infrastructure when the test is run.
The autotest infrastructure is maintained in the chromeos-base/autotest cros_workon ebuild.
Individual tests and test suites are in the chromeos-base/autotest-tests ebuild.

The new way: Use cros_workon with all Python tests

Working on all Python autotests should be simple and fast.
First cros_workon autotest for the board on which you plan to run tests:

(chroot)$ cros_workon start autotest --board=${BOARD}

Now, test_that will detect that you are working on autotests and automatically copy over Python in the autotest source tree to the sysroot.

Run test_that as usual:

(chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke

The old way: emerge individual autotest tests

If you have a test with binary dependencies, you must rebuild them explicitly:

(chroot)$ emerge-${BUILD} autotest-tests

This can take a very, very long time since it will rebuild ALL tests.
To speed this up, set the TESTS variable on the command line when runnning emerge to install an individual test:

(chroot)$ TESTS=<TestName> emerge-${BOARD} autotest-tests

Note that doing this will deactivate all other tests! This is a problem for test suites.
To rebuild a whole test suite, specify the suite and all individual tests in the suite.  For example, to rebuild smoke suite:

(chroot)$ TESTS='build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits' \
emerge-${BOARD} autotest-tests

Then go grab a coffee...

Debugging

You can use chromium os debugging tips on VM as well.

Comments