Building Factory Test Images

Introduction to factory setup

The Chromium OS reference factory software provides an example factory install and test flow for manufacturing of Chrome OS-based devices. It is available as part of the public Chromium OS repository on chromium.org. It is expected that this code will need to be modified on a per-product basis. The reference sequence is as follows:

  1. Read Only Firmware is programmed at SMT time by pulling the Firmware Write Protect pin low and flashing the firmware through an external device.
  2. After system assembly, the Factory Install Shim SD card is inserted. On boot, it will contact a Mini Omaha Server and request an install image. U-boot can support tftp boot/install from network without an SD card.
  3. A factory test image, a signed release image, and any other image data will be installed at this time. The system contains two full images: the test image and the shipping image.
  4. The system automatically reboots into the factory test image and begins manufacturing tests. This test suite is based on autotest and does not require network connectivity. The software supports sequencing tests, configuration, firmware and configuration updates, reboots, and other events in a configurable sequence.
  5. Functional tests, runin tests, and manual tests run as configured. Upon completion, results are displayed on the screen and are also available as an electronic pass/fail record plus detailed logs for upload at this point.
  6. The factory test image is automatically erased, leaving the release image as bootable.
  7. On a failure, the system can continue running further tests and report the failure on completion. Alternatively it can be configured to halt on failure at specific points.
  8. The factory image and test image can be composited into an SSD image and imaged onto the internal drive before assembly. On first boot, the machine will enter the above sequence at step 4 above, booting into the factory test image.

How to build the factory install shim

You will need a Chromium OS development environment, as set up here:

https://sites.google.com/a/chromium.org/dev/chromium-os/developer-guide

  1. From inside the cros-chroot (you should finish setup_board and select the target board):
    (cros-chroot) ~/trunk/src/scripts $ ./setup_board --board x86-generic --default
  2. Build all the packages, including the factory install shim package
    ./build_packages
  3. Build a special factory install shim image (with special kernel and rootfs):
    ./build_image factory_install
  4. The target URL to connect to the mini-omaha server is stored in the factory install shim image in the file first partition of image file (/dev/sd?1:dev_image/etc/lsb-factory). You can modify this by the following commands:
    mkdir /tmp/stateful
    sudo mount /dev/sdb1 /tmp/stateful
    sudo vi /tmp/stateful/dev_image/etc/lsb-factory
    sudo umount /tmp/stateful
  5. You can add / modify:
    CHROMEOS_RELEASE_BOARD=board_name_here
    CHROMEOS_AUSERVER=http://your.server.com:8080/update
  6. This will create an image called factory_install_shim.bin in your build directory (../builds/$BOARD/latest).  Install the image to your SD card or USB stick with the command:  
    sudo dd bs=4M if=/path/to/image/factory_install_shim.bin of=/dev/sdX \
    iflag=fullblock oflag=dsync
  7. Upon boot, the factory install shim will display a download status and will download the image from the server. On completion, it will reboot.  If you are using legacy firmware (not Chrome OS firmware), it may be necessary to remove the SD card to allow booting into the newly installed image.

    You may remove the SD card when the image starts downloading (displayed message will become green color)—it is not needed after that point.

How to build a factory test image

You will need a Chromium OS development environment, as set up here:

https://sites.google.com/a/chromium.org/dev/chromium-os/developer-guide

You will also need any files specific to the system you are manufacturing that may be checked into private repos. You may access your upcoming board repositories in a .repo/local_manifest.xml file, provided by Google when your board repositories and authentication are set up.

1. From inside the cros-chroot:
(cros-chroot)~/trunk/src/scripts $

2. Build all the packages, including the autotest package
./build_packages

3. Build a test image. This will create an image called chromiumos_test_image.bin in your build directory.
./build_image test

4. Build the factory toolkit. This will create a file called install_factory_toolkit.run.
./make_factory_toolkit.sh

5. Run the factory toolkit, patching chromiumos_test_image.bin.
~/trunk/src/build/images/daisy_spring/latest/install_factory_toolkit.run \
    /path/to/chromiumos_test_image.bin

 Now when you image a device with chromiumos_test_image.bin, the factory tests will be enabled.

How to boot your factory test image via USB

For development, it is possible to boot the factory test image from a USB stick, rather than using network install. If you care only about network install, you do not need to create a USB-stick copy.

  1. First, copy your image to USB storage (some /dev/sdX)
    cros flash usb:// chromiumos_factory_image.bin
  2. Then, on your device, switch to developer mode, enter the shell, and type:
    sudo chromeos-firmwareupdate --mode=todev
  3. With the USB storage inserted, hit Ctrl-U at the blue boot warning screen. You can also enter VT2 and install the image to ssd by the chromeos_install command.

    4. Note: This does not apply to CR48—that system would need firmware write protection to be disabled.

How to prepare a factory package set

Usually a factory installation will put the factory test image in the first slot of OS image partitions (#2 and #3), and the release image in the second slot (#4 and #5).

For branches prior to R16

The release image must be bootable from SSD, which is not the default option for disk images created by build_image.

Prior to R16, you need to manually resign the image to be an SSD image. To do that:

./bin/cros_sign_to_ssd --from=../path/to/image/chromiumos_release_image.bin

This will sign the image with developer keys such that it can be run with developer mode off, with developer key signed firmware.

For branches R16 and later

A generic factory package can be built from a release image and a factory image as follows:

(chroot) ~/trunk/src/scripts/make_factory_package.sh \
  --factory=/path/to/chromiumos_factory_image.bin \
  --release=/path/to/chromiumos_image.bin \
  --hwid=/path/to/hwid_bundle.sh

This will generate a miniomaha.conf, and a partition update .gz set. Note that it will erase any previous configuration and partition update sets.

Starting with R16, if you need to run the same configuration several times, or if you're trying to release some preconfigured setup for partners, make_factory_package also supports a --config mode:

(chroot) ~/trunk/src/make_factory_package.sh \
  --config=path/to/config_file.conf

The config file is simply a text file with all parameters you want to assign. The configuration file also supports some variables to help you assign image file source in a relative path; for example, $MFP_CONFIG_DIR refers to the directory of config file. See make_factory_package.sh --help for more information. If you need to set up multiple boards, use [board-name] to separate: use --board for the first board and then --subfolder for all remaining boards.

Example with variables and multiple configuration

x86-generic

--board x86-generic
--factory $MFP_CONFIG_DIR/../factory_test/chromiumos_factory_image.bin
--release $MFP_CONFIG_DIR/../release/*-generic_*.bin
--hwid $MFP_CONFIG_DIR/../hwid/hwid_bundle_generic_mp.sh


x86-generic_3g

--subfolder x86-generic_3g --board x86-generic_3g
--factory $MFP_CONFIG_DIR/../factory_test/chromiumos_factory_image.bin 
--release $MFP_CONFIG_DIR/../release/*-generic-3g_*.bin
--hwid $MFP_CONFIG_DIR/../hwid/hwid_bundle_generic_mp.sh

Things to note

  • After R16, firmware will be automatically extracted from the release image (--release), unless you specify a different updater (by --firmware path/to/updater) or none to skip overwriting firmware. Before R16, the --firmware parameter defaults to none. A value of none should be used only for development and testing, not for actual mass production in the factory.
  • The hwid_bundle.sh can be found in the hwid subfolder of your factory test image if you have access to the internal source tree, or from the factory bundle archive (zip or tbz). If you're simply trying some factory tests and do not need to run the finalization, you can use --hwid none to skip HWID stuff.
  • The factory test image, factory install shim, and release image must be compatible. Since boot, update, and partition formats are not finalized yet, this means that all images must be built from the same branch for the time being.

How to set up for Mini Omaha Server installation

You will need a Chromium OS development environment, as set up here:

https://sites.google.com/a/chromium.org/dev/chromium-os/developer-guide

You will need the factory test image and release image update package set as described above.

You will need python2.6, with module cherrypy 3.0. This can be installed running sudo apt-get install python-cherrypy3 on ubuntu.

To prepare packages and start mini-omaha immediately:

(chroot) ~/trunk/src/make_factory_package.sh \
--factory=/path/to/chromiumos_factory_image.bin \
--release=/path/to/chromiumos_image.bin \
--hwid=/path/to/hwid_bundle.sh \
--run_omaha

or
~/trunk/src/make_factory_package.sh --config path/to/config --run

The --run_omaha parameter tells make_factory_setup.sh to invoke miniomaha.py (the mini omaha server) after packages have been prepared successfully.

If you want to manually start the server, change the working directory to ~/trunk/src/platform/factory-utils/factory_setup.

  • To check if the server is configured correctly:
    python ./miniomaha.py --factory_config miniomaha.conf  --validate_factory_config
  • To start the server with prepared factory packages:
    python ./miniomaha.py --factory_config miniomaha.conf

Output is logged to stdout, the server displays incoming requests, and files are served.

Note: Before R17, the path and file name of the mini omaha server is different, and there's no --run. You need to:

./make_factory_packages --release ... # all the params for packages
cd ../platform/dev
python ./devserver.py --factory_config miniomaha.conf

How to build an SSD image

If you prefer to preimage the machines rather than image over the network, you can generate the disk image from a factory test image and a release image.

A generic factory package can be built from a release image and a factory image:

./make_factory_package.sh --diskimg=ssd_image.bin \
--factory=/path/to/chromiumos_factory_image.bin \
--release=/path/to/chromiumos_image.bin \
--hwid=/path/to/hwid_bundle.sh

You may image directly to a device, or to a .bin file.

--diskimg=XX
specifies the destination device or file
--sectors=XX
specifies the number of sectors in the bin file
--preserve
prevents wiping of the unused space for faster imaging

How to modify the factory test image or add test cases

The factory test image will run the autotest control file located at src/third_party/autotest/files/client/site_tests/suite_Factory/control (which is installed in /usr/local/autotest/client/site_tests/suite_Factory/control on DUT) . This file controls the sequence of autotest modules that are run. It reads a test_list file in the same folder.

Autotest control file documentation can be found here:

http://autotest.kernel.org/wiki/ControlHowto

Any SW action that needs to be taken in manufacturing can be staged as an autotest module, in the directory src/third_party/autotest/files/client/site_tests/.

Startup on boot is controlled by /etc/init/factory.conf and /etc/init/autotest.conf upstart scripts. Status is logged into /var/log/factory.log.
Comments