FAFT (Fully Automated Firmware Tests) is a collection of tests and related infrastructure that exercise and verify capabilities of Chrome OS. The features tested by FAFT are implemented through low-level software (firmware/BIOS) and hardware. FAFT evolved from SAFT (Semi-Automated Firmware Tests) and you can locate tests in the FAFT suite in the Autotest tree as directories with the prefix firmware_. The founding principles of FAFT are:
- Fully automated, no human intervention required
- Real test of physical hardware, like USB plug-in, Ctrl-D key press
- High test coverage of complicated verified boot flows
- Easy to integrate with existing test infrastructure, i.e. deploy to test lab, run remotely, continuous testing, etc.
To access some of these low-level capabilities, the tests require a servo board(servoV2 or servoV4). You connect the servo board directly to the test device to enable access to low-level system hardware interfaces, as well as staging areas for backup software (on a USB drive). During testing the tests may corrupt various states in the EC, firmware, and kernel to verify recovery processes. In these cases you can almost always use FAFT to restore the system to its original state.
The FAFT suite of tests can be invoked locally or remotely. This document describes how to set up the local configuration only. In remote configurations for automated test labs, a beagleboard device is required to enable communication between the device and servo board, and the test server.
The Chrome OS firmware/BIOS control (among other things) initial setup of the system hardware during the boot process. They are necessarily complicated, providing reliability against various corruption scenarios and security to ensure trusted software is controlling the system. Currently, the purpose of FAFT is to exercise EC firmware and BIOS firmware (such as U-Boot).
Setting up the Hardware
The most common hardware configuration for running FAFT includes:
- a test controller (your host workstation with a working chroot environment)
- the test device (a device that can boot Chrome OS)
- a servo board
- related connectors
The following photo shows the details of how to connect the servoV2 board to the test controller, test device, and network:
The following photo shows the details of how to connect the latest debug board, servoV4(Type A) to the test controller, test device, and network:
Preliminary Tasks
After the hardware components are correctly configured, prepare and install a test Chromium OS image:
- Build the binary (chromiumos_test_image.bin) with build_image test, or fetch the file from a buildbot.
- Load the test image onto a USB drive.
- Insert the USB drive into the servo board, as shown in the photo.
- Install the test image onto the internal disk by booting from the USB drive and running chromeos-install.
To run FAFT you use the test_that tool, which does not automatically start a servod process for communicating with the servo board. Before running any tests:
- Run $ sudo servod --board=$BOARD where $BOARD is the code name of the board you are testing. For example: $ sudo servod --board=lumpy
- Run the firmware_FAFTSetup test to verify basic functionality and ensure that your setup is correct. If test_that is in /usr/bin, the syntax is
$ /usr/bin/test_that --board=$BOARD $REMOTE_IP firmware_FAFTSetup
Running Test Suites
To run FAFT on a Chromebook:
- Run the entire faft_bios suite:
$ /usr/bin/test_that --board=$BOARD $REMOTE_IP suite:faft_bios
- Run the entire faft_ec suite:
$ /usr/bin/test_that --board=$BOARD $REMOTE_IP suite:faft_ec
To run FAFT on a Chromebox or Chromebase, a USB-KM232 cable and a RS232-USB cable are required.
- Connect the USB-KM232 to the RS232-to-USB cable, then connect the RS232 side to the test controller.
- Connect the USB side to the Chrome device.
- Check the device on the host by running tail -f /var/log/messages. It should be named /dev/ttyUSB0.
- Assign this value to the USBKM232_UART_DEVICE environment variable and make the drive partition writeable. For example:
$ export USBKM232_UART_DEVICE=/dev/ttyUSB0
$ sudo chmod a+rw $USBKM232_UART_DEVICE
- When starting servod, specify the --usbkm232 argument.
$ sudo servod --board=$BOARD --usbkm232=$USBKM232_UART_DEVICE
- The test_that command line stays the same:
$ /usr/bin/test_that --board=$BOARD $REMOTE_IP f:.*DevMode/control$
FAQ
Q: All of my FAFT tests are failing. What should I do?
A: Run firmware_FAFTSetup as a single test. Once it fails, check the log and determine which step failed and why.
Q: A few of my FAFT tests failed, but most tests are passing. What should I do?
A: Re-run the failed tests by themselves and they may pass. Sometimes tests fail due to flaky infrastructure or other non-firmware bugs.
Q: I still need help. Who can help me?
Q: I got an error while running FAFT: AutoservRunError: command execution error: sudo -n which flash_ec . What's wrong?
A: Run sudo emerge chromeos-ec inside your chroot.
