For Testers‎ > ‎

FAFT

FAFT (Fully Automated Firmware Tests) is a collection of tests and related infrastructure that exercise and verify capabilities of Chrome OS. The features tested by FAFT are implemented through low-level software (firmware/BIOS) and hardware. FAFT evolved from SAFT (Semi-Automated Firmware Tests) and you can locate tests in the FAFT suite in the Autotest tree as directories with the prefix firmware_. The founding principles of FAFT are:

  • Fully automated, no human intervention required
  • Real test of physical hardware, like USB plug-in, Ctrl-D key press
  • High test coverage of complicated verified boot flows
  • Easy to integrate with existing test infrastructure, i.e. deploy to test lab, run remotely, continuous testing, etc.
The firmware/bios of ChromeOS control (among other things) initial setup of the system hardware during boot.  They are complicated due to the intent of adding reliability against various corruption scenarios and security that ensures trusted software is controlling the system.  Currently, FAFT tests exist to exercise ec firmware and bios firmware (such as U-Boot).

Much of the FAFT testing uses a common pattern called a FAFT-step which includes each of the following operations:
  • check system state (such as low-level flags related to manner in which the device booted)
  • run any set of user-space actions to setup the test (including modifying system state to force variations in the next boot)
  • force a system to reboot (or into recovery via soft-reboot, hard-reboot, cold-reboot, etc.)
  • script some firmware actions during the following boot-up (to simulate key-presses such as <Ctrl>+d during boot flow)
Most FAFT tests are a series of such steps called a FAFTSequence.  FAFT sequences are controlled from the test server and communicate with the test device (DUT) to run device commands.  FAFT libraries exist (server/cros/faftsequence.py, server/cros/faft_client_attributes.py, client/cros/faft_client.py, client/cros/saft) to wrap common device functions and interfaces for use in FAFT testing. 

To access some of these low-level capabilities, the tests require a servo board(servoV2 or servoV4). You connect the servo board directly to the test device to enable access to low-level system hardware interfaces, as well as staging areas for backup software (on a USB drive). During testing the tests may corrupt various states in the EC, firmware, and kernel to verify recovery processes. In these cases you can almost always use FAFT to restore the system to its original state.

The FAFT suite of tests can be invoked locally or remotely. This document describes how to set up the local configuration only. In remote configurations for automated test labs, a beagleboard device is required to enable communication between the device and servo board, and the test server.

The Chrome OS firmware/BIOS control (among other things) initial setup of the system hardware during the boot process. They are necessarily complicated, providing reliability against various corruption scenarios and security to ensure trusted software is controlling the system. Currently, the purpose of FAFT is to exercise EC firmware and BIOS firmware (such as U-Boot).

Setting up the Hardware

The most common hardware configuration for running FAFT includes:
  • a test controller (your host workstation with a working chroot environment)
  • the test device (a device that can boot Chrome OS)
  • a servo board
  • related connectors


The following photo shows the details of how to connect the servoV2 board to the test controller, test device, and network:

Details of servoV2 connections:
  1. Connect one end(ribbon cable) of the flex cable to servoV2 and the other end to the debug header on the chrome device.
  2. Connect DUT_HUB_IN(micro USB port) of the servo to the DUT.
  3. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown the diagram.
  4. Connect the micro USB port of the servo to the host machine(workstation or a labstation).
  5. Connect an ethernet cable to the ethernet jack of the servo.

The following photo shows the details of how to connect the latest debug board, servoV4(Type A) to the test controller, test device, and network:

Details of servoV4 connections:
  1. Connect one end(micro USB) of the servo micro flex cable to servoV4 using micro USB to USB cable. Connect the other end to the debug header on the chrome device.
  2. Connect the USB type A cable of the servoV4 to the DUT.
  3. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown the diagram.
  4. Connect the micro USB port of the servo to the host machine(workstation or a labstation).
  5. Connect an ethernet cable to the ethernet jack of the servo.

Preliminary Tasks

After the hardware components are correctly configured, prepare and install a test Chromium OS image:

  1. Build the binary (chromiumos_test_image.bin) with build_image test, or fetch the file from a buildbot.
  2. Load the test image onto a USB drive.
  3. Insert the USB drive into the servo board, as shown in the photo. 
  4. Install the test image onto the internal disk by booting from the USB drive and running chromeos-install.
To run FAFT you use the test_that tool, which does not automatically start a servod process for communicating with the servo board. Before running any tests:
  1. Run $ sudo servod --board=$BOARD where $BOARD is the code name of the board you are testing. For example: $ sudo servod --board=lumpy 
  2. Run the firmware_FAFTSetup test to verify basic functionality and ensure that your setup is correct. If test_that is in /usr/bin, the syntax is $ /usr/bin/test_that --board=$BOARD $REMOTE_IP firmware_FAFTSetup

Running Test Suites

To run FAFT on a Chromebook:

  1. Running FAFT test with test case name
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP f:.*DevMode/control

  2. Some tests can be run in either normal mode or dev mode, specify the control file
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP f:.*TryFwB/control.dev

  3. FAFT can install Chrome OS image from the USB when image filename is specified
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP --args "image=$IMAGE_FILE" f:.*RecoveryButton/control.normal

  4. To update the firmware using the shellball in the image, specify the argument firmware_update=1
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP --args "image=$IMAGE_FILE firmware_update=1" f:.*RecoveryButton/control.normal

  5. Run the entire faft_bios suite:
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP suite:faft_bios

  6. Run the entire faft_ec suite
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP suite:faft_ec
To run FAFT on a Chromebox or Chromebase, a USB-KM232 cable and a RS232-USB cable are required.
  1. Connect the USB-KM232 to the RS232-to-USB cable, then connect the RS232 side to the test controller. 

  2. Connect the USB side to the Chrome device.

  3. Check the device on the host by running tail -f /var/log/messages.  It should be named /dev/ttyUSB0. 

  4. Assign this value to the USBKM232_UART_DEVICE environment variable and make the drive partition writeable. For example:
    $ export USBKM232_UART_DEVICE=/dev/ttyUSB0
    $ sudo chmod a+rw $USBKM232_UART_DEVICE

  5. When starting servod, specify the --usbkm232 argument. 
    $ sudo servod --board=$BOARD --usbkm232=$USBKM232_UART_DEVICE 

  6. The test_that command line stays the same:
    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP f:.*DevMode/control
To run servod in a different host (like using Beaglebone), specify the servo_host and servo_port arguments.

    $ /usr/bin/test_that --board=$BOARD $REMOTE_IP --args "servo_host=$SERVO_HOST servo_port=$SERVO_PORT" suite:faft_lv1
 

FAQ

Q: All of my FAFT tests are failing. What should I do?
A: Run firmware_FAFTSetup as a single test. Once it fails, check the log and determine which step failed and why.

Q: A few of my FAFT tests failed, but most tests are passing. What should I do?
A: Re-run the failed tests by themselves and they may pass. Sometimes tests fail due to flaky infrastructure or other non-firmware bugs.

Q: I still need help. Who can help me?
A: Try joining the FAFT-users chromium.org mailing list and asking for help. Be sure to include logs and test details in your request for help.

Q: I got an error while running FAFT: AutoservRunError: command execution error:  sudo -n which flash_ec . What's wrong?
A: Run sudo emerge chromeos-ec inside your chroot.
Comments