Introduction

Target audience

Organization & content

Typography conventions

• Paths, files, and commands are shown in different colors to indicate whether they apply to (1) your build computer (the computer on which you're doing development), (2) the chroot on your build computer, or (3) your Chromium OS computer (the device on which you run the images you build):

Color	Paths, files, and commands:
green text	on your build computer
purple text	inside the chroot on your build computer
crimson text	on your Chromium OS computer

• Notes are shown using the following conventions:
  • IMPORTANT NOTE describes required actions and critical information
    
  • SIDE NOTE describes explanations, related information, and alternative options
    
  • TODO describes questions or work that is needed on this document

Modifying this document

• If you see a red TODO and you know the right answer, fix it!
• If you see something wrong, fix it. 
• If you're not sure of the perfect answer, still fix it.   Stick in a red TODO noting your uncertainty if you aren't sure, but don't let anything you know to be wrong stick around.

• Put all general "getting started" information directly on this page. It's OK if this document grows big.
• If some information is also relevant for other documents, put the information in this document and add links from the other documents to this document. If you do not have a choice and must put information relevant to getting started in other documents, add links in this document to discrete chunks of information in the other documents. Following a web of links can be very confusing, especially for people who are new to a project, so it's very important to maintain a linear flow as much as possible.
  
• Where possible, describe a single way of doing things. If you specify multiple options, provide a clear explanation why someone following the instructions would pick one option over the others. If there is disagreement about how to do things, consider listing alternative options in a SIDE NOTE.
• Keep Google-specific references to a minimum. A large number of the people working on Chromium OS work at Google, and this document may occasionally contain references that are only relevant to engineers at Google. Google engineers, please keep such references to a minimum – Chromium OS is an open source project that should stay as open as possible.

Additional information

Get the source code

Googlers: See goto/chromeos-building for internal notes.

Optionally add Google API keys

Branch Builds

If you want to build on a branch, pass the branch name to repo init (e.g: repo init -u <URL> [-g minilayout] -b 0.9.94.T).

When you use repo init you will be asked to confirm your name, email address, and whether you want color in your terminal. This command runs quickly. The repo sync command takes a lot longer.

More info can be found in the working on a branch page.

Installing Chromium OS on your Device

Put your image on a USB disk

The easiest way to get your image running on your target computer is to put the image on a USB flash disk (sometimes called a USB key), and boot the target computer from the flash disk. The first step is to insert a USB flash disk (4GB or bigger) into your build computer. This disk will be completely erased, so make sure it doesn't have anything important on it.  Wait ~10 seconds for the USB disk to register, then type the following command:

For more details on using this tool, see the Cros Flash page. Note that auto-mounting of USB devices should be turned off as it may corrupt the disk image while it's being written.

Disable verified boot

You need to disable verified boot to be able to use your own image. Sometimes it can be achieved by holding the reset shortcuts (Esc-F3-Power) for a long time. Try with your machine for a while until you figure it out. Disabling verified boot takes some time, and is often accompanied by entering developer mode.

How this is accomplished for non Google Chrome OS devices depends heavily on the BIOS and can vary drastically, so we will not attempt to cover that.

For Google Chrome OS devices, see the Developer Hardware page. (Pages of each devices will show how to enter dev mode which will disable verified boot.)

Boot from your USB disk

Now reboot. In the white screen (developer mode enabled) screen, plug-in the USB and press Ctrl+U (or whatever key boots from USB on your device).

For specific information about what works on various different machines, see the Developer Hardware page.

Installing your Chromium OS image to your hard disk

Getting to a command prompt on Chromium OS

1. Go through the standard Chromium OS login screen (you'll need to setup a network, etc.) and get to the web browser.  It's OK to login as guest.
2. Press [ Ctrl ] [ Alt ] [ T ] to get the crosh shell.
3. Use the shell command to get the shell prompt. NOTE: you don't need to enter the chronos password here, though you will still need the password if you want to use the sudo command.

---

---

Building an image to run in a virtual machine

Many times it is easier to simply run Chromium OS in a virtual machine like kvm.  You can adapt the previously built Chromium OS image so that it is usable by kvm (which uses qemu images) by entering this command from the ~/trunk/src/scripts directory:

./image_to_vm.sh --board=${BOARD}

This command creates the file ~/trunk/src/build/images/${BOARD}/latest/chromiumos_qemu_image.bin.

SIDE NOTES:

• WARNING: After crbug/710629, 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so is the most likely to work at ToT. 'betty' is based on 'amd64-generic', though, so 'amd64-generic' is likely to also work.
• Only KVM/QEMU VM's are actively supported at the moment.
• You can specify source/destination paths with the --from and --to parameters.
• If you're interested in creating a test image (used for allowing Chromium OS to talk to autotest), see the Running Tests section.

---

---

Making changes to packages whose source code is checked into Chromium OS git repositories

Now that you can build and run Chromium OS, you're ready to start making changes to the code.

NOTE: If you skipped to this section without building your own system image, you may run into hard-to-fix dependency problems if build your own versions of system packages and try to deploy them to a system image that was built by a builder. If you run into trouble, try going through the full Building Chromium OS process first and installing your own system image.

---

Keep the tree green

Before you start, take a moment to understand Chromium's source management strategy of "keeping the tree green". For the Chromium OS project, keeping the tree green means:

1. Any new commits should not destabilize the build:
• Images built from the tree should always have basic functionality working.
• There may be minor functionality not working, and it may be the case, for example, that you will need to use Terminal to fix or work around some of the problems. 
  
  
2. If you must introduce unstable changes to the tree (which should happen infrequently), you should use parameterization to hide new, unstable functionality behind a flag that's turned off by default. The Chromium OS team leaders may need to develop mechanisms for parameterizing different parts of the system (such as the init script).
   
   
3. Internal "dev channel" releases will be produced directly from the tree, with a quick branch to check-point the release version. Any fixes required for a release will be pulled from the tree, avoiding merges back to tree.

This strategy has many benefits, including avoiding separate build trains for parallel development (and the cost of supporting such development), as well as avoiding large, costly merges from forked branches.

SIDE NOTE: "Keep the tree green" means something a bit different for Chromium OS than for Chromium, which is much further along in its life cycle. 

The steps in this section describe how to make changes to a Chromium OS package whose source is checked into the Chromium OS source control system. Specifically, this is a package where:

• The ebuild for the package lives in the src/third_party/chromiumos-overlay or src/overlays/overlay-${BOARD} directories.
• There is an ebuild for the package that ends with 9999.ebuild.
• The ebuild inherits from the cros-workon class.
• The ebuild has a KEYWORDS in the ebuild containing this architecture name (like "x86").

You can see a list of all such packages by running the following command from inside the ~/trunk/src/scripts directory:

cros_workon --board=${BOARD} list --all


---

Run cros_workon start

The first thing you need to do is to mark the package as active. Use the command below, replacing ${PACKAGE_NAME} with your package name (e.g., chromeos-wm):

cros_workon --board=${BOARD} start ${PACKAGE_NAME}

This command:

• Indicates that you'd like to build the 9999 version of the ebuild instead of the stable, committed version.
• Indicates that you'd like to build from source every time.
• If you specified that you wanted the minilayout when you did your repo init, this command adds a clause to your .repo/local_manifest.xml to tell repo to sync down the source code for this package next time you do a repo sync.

---

Run repo sync

After running cros_workon, sync down the sources. This is critical if you're using the minilayout, but is probably a good idea in any case to make sure that you're working with the latest code (it'll help avoid merge conflicts later).  Run the command below from outside the chroot, anywhere under your ~/chromiumos directory:

repo sync

---

Find out which ebuilds map to which directories

The cros_workon tool can help you find out what ebuilds map to each directory. You can view a full list of ebuilds and directories using the following command:

cros_workon info --board=${BOARD} --all

If you want to find out which ebuilds use source code from a specific directory, you can use grep to find them. For example:

cros_workon info --board=${BOARD} --all | grep platform/ec

This returns the following output:

    chromeos-base/ec-utils chromiumos/platform/ec src/platform/ec

This tells you the following information:

1. The name of the ebuild is chromeos-base/ec-utils
2. The path to the git repository on the server is chromiumos/platform/ec
3. The path to the source code on your system is src/platform/ec

You can similarly find what source code is associated with a given ebuild by grepping for the ebuild name in the list.

To find out where the ebuild lives:

equery-${BOARD} which ${PACKAGE_NAME}

As an example, for PACKAGE_NAME=ec-utils, the above command might display:
    /home/.../trunk/src/third_party/chromiumos-overlay/chromeos-base/ec-utils/ec-utils-9999.ebuild

SIDE NOTE: If you run the same command without running cros_workon first, you can see the difference:

    /home/.../trunk/src/third_party/chromiumos-overlay/chromeos-base/ec-utils/ec-utils-0.0.1-r134.ebuild

---

Create a branch for your changes

Since Chromium OS uses repo/git, you should always create a local branch whenever you make changes.

First, find the source directory for the project you just used cros_workon on. This isn't directly related to the project name you used with cros_workon. (This isn't very helpful - someone with more experience, actually tell us how to find it reliably? --Meredydd)

cd into that directory, in particular the "files/" directory in which the actual source resides. In the command below, replace ${BRANCH_NAME} with a name that is meaningful to you and that describes your changes (nobody else will see this name):

repo start ${BRANCH_NAME} .

The branch that this creates will be based on the remote branch (which one? --Meredydd). If you've made any other local changes, they will not be present in this branch.

---

Make your changes

You should be able to make your changes to the source code now.  To incrementally compile your changes, use 

cros_workon_make --board=${BOARD} ${PACKAGE_NAME}

This will build your package inside your source directory.  Change a single file, and it will rebuild only that file and re-link.  If your package contains test binaries, using

cros_workon_make --board=${BOARD} ${PACKAGE_NAME} --test

will build and run those binaries as well.  Call cros_workon_make --help to see other options that are supported.

You probably want to get your changes onto your device now.  You need to install the changes you made by using 

cros_workon_make --board=${BOARD} ${PACKAGE_NAME} --install

You can then rebuild an image with build_image and reimage your device, but you might also want to take a peek at cros deploy if you want something a little quicker.

---

Set your editor

Many of the commands below (in particular git) open up an editor. You probably want to run one of the three commands below depending on your favorite editor. 

If you're not a *nix expert, nano is a reasonable editor:

export EDITOR='nano'

If you love vi:

export EDITOR='vi'

If you love emacs (and don't want an XWindow to open up every time you do something):

export EDITOR='emacs -nw'

You should probably add one of those lines to your .bashrc (or similar file) too.

---

Submit changes locally

When your changes look good, commit them to your local branch using git. Full documentation of how to use git is beyond the scope of this guide, but you might be able to commit your changes by running something like the command below from the project directory:

git commit -a

The git commit command brings up a text editor. You should describe your changes, save, and exit the editor. Note that the description you provide is only for your own use. When you upload your changes for code review, the repo upload command grabs all of your previous descriptions, and gives you a chance to edit them.

---

Upload your changes and get a code review

Once your changes are committed locally, upload your changes using repo upload. The repo upload command takes all of the changes that are unmerged and asks them if you want to upload them.  You can specifically say to only look for unmerged changes in your current repo by passing in '.'.  Please note that you must have a Gerrit account before you can upload changes.

repo upload [.|${PROJECT-NAME}] [--current-branch]

It is important to note that repo uses the Change-Id in your git commits to track code reviews.  So in order to work on a CL the standard work flow is to use git commit --amend rather than make a new commit.  This differs from our old git-cl style.

In your local commit logs make sure to add a BUG= field and TEST= field.  For BUG=, you should put something that looks like: BUG=bug-tracker:number. You can get a tracker name by looking in the upper-left corner of the tracker page (e.g. in the image on the right, the tracker name is "chromium-os"). If your changes are related to more than one tracker issue, you can list all the issues separated with commas.

For TEST=, you should describe what you did to test the changes.

Here's what a sample description should look like:

# Enter a description of the change.

# This will displayed on the codereview site.

# The first line will also be used as the subject of the review.

Here's a SHORT, one-line summary of my change.

And here are more details

...this can be as long as I want.

BUG=chromium-os:99999, chromium:88888

TEST=Ran all the white box tests




Change-Id: I8d7f86d716f1da76f4c85259f401c3ccc9a031ff

Once you run repo upload, this uploads the changes and prints out a URL for the code review (if it's a new code review). Go to that URL (log in with your chromium.org account, which you might want to do in an "incognito" window in Chrome), and use the "Review->Publish Comments" link to mail your changes to your reviewers.

You should pick reviewers that know the code you're working on well and that will do the best reviews. Picking reviewers who will just rubber-stamp your changes is a bad idea. The point of submitting changes is to submit good code, not to submit as much code as you can. If you don't know who should review your changes, start by looking at the git log for the project that you're working on. Simply type the command below in a directory related to your project:

git log

Your reviewers will likely provide comments about changes that you should make before submitting your code. You should make such changes, submit them locally, and then re-upload your changes for code review by amending your changes to your git commit and re-running repo upload.

# make some changes

git add -u .

git commit --amend

If you have a chain of commits (which repo upload . converts to a chain of CLs), and you need to modify any commits that are not at the top of the chain, use interactive rebase:

git rebase -i

# This shows a list of cherry-picks into a temporary branch.

# Change some of the "pick" keywords to "edit".  Then exit the editor.

git log # shows you are at the first "edit"ed commit.  All earlier commits are cherry-picked.

# Make some modifications.

git add -u .

git commit --amend

git rebase --continue # goes to the next "edit"ed commit.

repo upload . --current-branch # when all modifications are ready to be uploaded again.

---

Try seeing the mainline again

While you're working on your changes, you might want to go back to the mainline for a little while (maybe you want to see if some bug you are seeing is related to your changes, or if the bug was always there). If you want to go back to the mainline without abandoning your changes, you can run the following command from within a directory associated with your project:

git checkout cros/master

When you're done, you can get back to your changes by running:

git checkout ${BRANCH_NAME} 

---

Work on something else while you're waiting

If you want to start on another (unrelated) change while waiting for your code review, you can repo start another branch. When you want to get back to your first branch, run the following command from within a directory associated with your project:

git checkout ${BRANCH_NAME} 

---

Send your changes to the Commit Queue

Eventually, all your reviewers will be happy and will give you a Looks Good and Approved (the latter is only available to owners of the code) message in Gerrit.  When a reviewer is satisfied with the CL but wants other reviewers to approve they may give a +1. If they choose to test it they can also mark Verified but typically you'll need to test the CL and mark Verified yourself.  Both Looks Good and Approved (+2) and Verified must be set in order to commit your CL.  Once they are set, set the Commit Ready bit on your CL and the commit queue will test the change.

Before the Commit Queue picks up your CL it must pass a pre-CQ trybot.  This trybot run is triggered automatically when your CL is marked Looks Good and Approved.  You can trigger this trybot earlier by checking 'Trybot-Ready' and, if the run passes, it will not be required again before the Commit Queue picks up the CL.

Note it is possible that your change will be rejected because of a merge conflict.  If it is, rebase against any new changes and re-upload your patch.  This patch will have to be re-approved before it is allowed to be committed. (Note: If it's just a trivial rebase, you can approve the rebase yourself.)

More details on the Commit Queue can be found in the Commit Queue Overview.

---

Make sure your changes didn't break things

After you commit, make sure you didn't break the build by checking the buildbot.

---

Share your changes using the Gerrit sandbox

It is possible to upload changes to a personal sandbox on Gerrit. This way, a change can be shared between developers before it is ready for code review.

project_url=https://chromium.googlesource.com/$(git config remote.cros.projectname)
git push ${project_url} HEAD:refs/sandbox/${USER}/${BRANCH_NAME}

Other developers can then fetch your changes using the following commands:

project_url=https://chromium.googlesource.com/$(git config remote.cros.projectname)

git fetch ${project_url} refs/sandbox/${USER}/${BRANCH_NAME}
git checkout FETCH_HEAD

In a given repository, you can explore sandboxes using the ls-remote command:

git ls-remote cros "refs/sandbox/${USER}/*"
git ls-remote cros "refs/sandbox/*"

Once you're finished with a sandbox, you can delete it using the following commands:

project_url=https://chromium.googlesource.com/$(git config remote.cros.projectname)
git push $project_url :refs/sandbox/${USER}/${BRANCH_NAME}

---

Clean up after you're done with your changes

After you're done with your changes, you're ready to clean up. The most important thing to do is to tell cros_workon that you're done by running the following command:

cros_workon --board=${BOARD} stop ${PACKAGE_NAME}

This command tells cros_workon to stop forcing the -9999.ebuild and to stop forcing a build from source every time.

You can also delete the branch that repo created. There are a number of ways to do so; here is one way:

repo abandon ${BRANCH_NAME} ${CROS_WORKON_PROJECT}

SIDE NOTES:

• If you don't specify a project name, the repo abandon command will throw out any local changes across all projects. You might also want to look at git branch -D or repo prune.
• If you're using the minilayout, doing a cros_workon stop will not remove your source code. The code will continue to stay on your hard disk and get synced down.

---

---

Making changes to non-cros_workon-able packages

If you want to make to changes to something other than packages whose source is checked into the Chromium OS source control system, you can follow the instructions in the previous section, but skip the cros_workon step. Note specifically that you still need to run repo start to create a branch for your changes.

The types of changes that fall into this category include:

• changes to build scripts (pretty much anything in src/scripts)
• changes to ebuild files themselves (like the ones in src/third_party/chromiumos-overlay)
  
• To change these files, you need to "manually uprev" the package. For example, if we're making a modification to for the Pixel overlay (link), then 
• cd src/overlays/overlay-link/chromeos-base/chromeos-bsp-link
• mv chromeos-bsp-link-0.0.2-r29.ebuild chromeos-bsp-link-0.0.2-r30.ebuild
• chromeos-bsp-link-0.0.2-r29.ebuild is a symlink that points to chromeos-bsp-link-0.0.2.ebuild. To uprev the package, simply increment the revision (r29) number.
• adding small patches to existing packages whose source code is NOT checked into Chromium OS git repositories
• changes to eclass files (like the ones in src/third_party/chromiumos-overlay/eclass)
• changes to the buildbot infrastructure (in crostools)
• TODO: anything else?

---

Adding small patches to existing packages

When you need to add small patches to existing packages whose source code is not checked into a Chromium OS git repository (e.g. it comes from portage, and is not a cros_workon-able package), you need to do the following:

First, find the package ebuild file under third_party/chromiumos-overlay.

Then, create a patch file from the exact version of the package that is used by the current ebuild.  If other patches are already in the ebuild, you'll want to add your patch LAST, and build the patch off of the source that has already had the existing patches applied (either do it by hand, or set FEATURES=noclean and build your patch off of the temp source).  Note that patch order is significant, since the ebuild expects each patch line number to be accurate after the previous patch is applied.

Place your patch in the "files" subdir of the directory that contains the ebuild file (e.g. third_party/chromiumos-overlay/dev-libs/mypackage/files/mypackage-1.0.0-my-little-patch.patch).

Then, in the prepare() section of the ebuild (create one if it doesn't exist), add an epatch line:

epatch "${FILESDIR}"/${P}-my-little-patch.patch

Lastly, you'll need to bump the revision number in the name of the ebuild file (or symlink) so the build system picks up the change.  The current wisdom is that the ebuild file should be symlinked instead of being renamed.  For example, if the original ebuild file is "mypackage-1.0.0.ebuild", you should create a "mypackage-1.0.0-r1.ebuild" symbolic link that points at the original ebuild file.  If that symlink already exists, create the next higher "rN" symlink.

---

Making changes to the way that the chroot is constructed

TODO: This section is currently a placeholder, waiting for someone to fill it in.  However, a few notes:

• Many of the commands that take a --board=${BOARD} parameter also take a --host parameter, which makes the commands affect the host (i.e. the chroot) rather than the board.
• Most notably, cros_workon --host says that you want to build a package used in the chroot from source.

---

Building an individual package

TODO: Document this better, and add the new cros_workon_make.

SIDE NOTE:To build an individual portage package, for a particular board, use emerge-${BOARD}.

For example, if you want to build dash to test on your device:

emerge-${BOARD} dash

To install the package to the device, see cros deploy.

---

Making changes to the Chromium web browser on Chromium OS

If you just want to make modifications to the Chromium web browser and quickly deploy your changes to an already-built Chromium OS image, see Making changes to the Chromium web browser on Chromium OS.

To use your local checkout of the Chromium source code when building a Chromium OS image, set the --chrome_root flag appropriately when entering the chroot, e.g.

cros_sdk --chrome_root=${HOME}/chrome

Within the chroot, you'll also need to either start working on the chromeos-chrome package:

cros_workon --board=${BOARD} start chromeos-chrome

or set the CHROME_ORIGIN environment variable appropriately:

export CHROME_ORIGIN=LOCAL_SOURCE

See src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild for additional possible values for the CHROME_ORIGIN variable.

If you have an internal checkout of the Google Chrome source and want to build the browser with official branding, export USE=chrome_internal.

---

---

Using Clang to get better compiler diagnostics

---

Description

The ChromeOS toolchain provides a feature to get Clang syntax-only compiler diagnostics without having to do a separate build with the Clang compiler. To enable this feature, add '-clang' to the C[XX]FLAGS used by the package for its build. 

Addition of the '-clang' option to the build is interpreted by the compiler driver wrapper script, which then invokes Clang with -fsyntax-only option, and after a successful Clang run, invokes the gcc compiler. Any errors generated by Clang will stop compilation just like a regular build does. In addition to Clang warnings, you will also see warning from gcc, in some cases for the same source construct. 

The presence of a few specific gcc options, for example, '-print-*' or '-E' will disable a clang run, even if '-clang' is specified. This is to allow package configure scripts to run correctly even in the presence of the '-clang' option. 

---

Wrapper Options

The wrapper script also interprets a few other options. All options specific to the wrapper only are tabulated below:

Option	Description
-clang	Invoke Clang front-end with -fsyntax-only and all other options specified on the command line. On successful completion of Clang compile, continue the build with gcc or g++.  The presence of '-print-*', '-dump*', '@*', '-E', '-' or '-M' will disable clang invocation.
-Xclang-only=<option>	This is a special option that can be used to pass <option> to Clang and not to gcc or g++. This can be used, for example, to turn off a specific Clang warning. Example: -Xclang-only=-Wno-c++11-extensions will add -Wno-c++11-extensions to the Clang invocation.
-print-cmdline	In addition to doing the builds, print the exact command-line used for both Clang and gcc.

---

Testing

You can test your package with Clang before adding '-clang' to your ebuild or Makefiles using the CFLAGS or CXXFLAGS variable. While using this, you need to be careful not to overwrite existing CFLAGS or CXXFLAGS. Here's an example: 

(cr) $ CFLAGS="$(portageq-$board envvar CFLAGS) -clang" CXXFLAGS="$(portageq-$board envvar CXXFLAG) -clang" emerge-$board chromeos-chrome




After your package builds cleanly with Clang, you can add '-clang' to your e-build. 

---

---

Local Debugging

---

Debugging both x86 and non-x86 binaries on your workstation.

If you build your projects incrementally, write unit tests and use them to drive your development, you may want to debug your code without shipping it over to a running device or VM.

gdb-${BOARD} sets up gdb in your board sysroot and ensures that gdb is using the proper libraries, debug files, etc. for debugging, allowing you to run your target-compiled binaries. 

It should already be installed in your chroot.  If you do not have the script, update your repository to get the latest changes, then re-build your packages:

repo sync

./build_packages --board=...


This should install gdb-${BOARD} in the /usr/local/bin directory inside the chroot.   These board-specific gdb wrapper scripts correctly handle both local and remote debugging (see next section for more information on remote debugging).  When used for local debugging, these scripts will run inside a special chroot-inside-your-chroot, rooted in the board's sysroot. For example if you are using gdb-lumpy, it will run inside a chroot based entirely in your /build/lumpy sysroot.  The libraries that it will load and use are the libraries in the sysroot, i.e. the target libraries for the board; the gdb binary it will use is the gdb  binary in that tree.  While debugging with gdb-lumpy (for local debugging), you will not be able to see/access any files outside of the /build/lumpy tree.  While for the most part this is very good, as it ensures the correct debugging environment, it does mean that if you want to use this script to debug a lumpy binary, such as a unit test, that you built outside of the /build/lumpy tree, you will need to copy the binary to the /build/lumpy tree first.  Also, if you want the debugger to be able to see the source files when debugging, you will need to make sure they exist inside the /build/lumpy tree as well (see example below).

IMPORTANT NOTE 1:  Local and remote debugging functionality are combined in this single script.  Some of the options shown below only work for remote debugging.

IMPORTANT NOTE 2:  When doing local debugging of x86 binaries, they will try to execute on your desktop machine (using the appropriate libraries and gdb binaries).  It is possible that for some x86 boards, the binaries may use instructions not understood by your hardware (particularly some vector instructions), in which case you will need to do remote debugging with the actual hardware instead.

IMPORTANT NOTE 3:  You can use this script with *some* debugging functionality for local debugging of non-x86 binaries.  The script loads qemu and runs the non-x86 binaries in qemu.  However qemu

has some unfortunate limitations.  For example you can "set" breakpoints in the binary (to see what addresses correspond to locations in the source), examine the source or assembly code, and execute the program.  But qemu does not actually hit the breakpoints, so you cannot suspend execution in the middle when running under qemu.   For full debugging functionality with non-x86 binaries, you must debug them remotely running on the correct hardware (see next section on remote debugging).  You can see this in the example below, where gdb-daisy does not actually stop at the breakpoint it appears to set, although it does correctly execute the program.

(cr) $ gdb-daisy -h




usage: cros_gdb [-h]

                [--log-level {fatal,critical,error,warning,notice,info,debug}]

                [--log_format LOG_FORMAT] [--debug] [--nocolor]

                [--board BOARD] [-g GDB_ARGS] [--remote REMOTE] [--pid PID]

                [--remote_pid PID] [--no-ping] [--attach ATTACH_NAME] [--cgdb]

                [binary-to-be-debugged] [args-for-binary-being-debugged]

Wrapper for running gdb.

This handles the fun details like running against the right sysroot, via

qemu, bind mounts, etc...

positional arguments:

  inf_args              Arguments for gdb to pass to the program being

                        debugged. These are positional and must come at the

                        end of the command line. This will not work if

                        attaching to an already running program.

...

(cr) $ gdb-daisy /bin/grep shebang /bin/ls

15:51:06: INFO: RunCommand: file /build/daisy/bin/grep

Reading symbols from /bin/grep...Reading symbols from /usr/lib/debug/bin/grep.debug...done.

done.

(daisy-gdb) b main

Breakpoint 1 at 0x2814: file grep.c, line 2111.

(daisy-gdb) disass main

Dump of assembler code for function main:

   0x00002814 <+0>: ldr.w r2, [pc, #3408] ; 0x3568 <main+3412>

   0x00002818 <+4>: str.w r4, [sp, #-36]!

   0x0000281c <+8>: movs r4, #0

   0x0000281e <+10>: strd r5, r6, [sp, #4]

   0x00002822 <+14>: ldr.w r3, [pc, #3400] ; 0x356c <main+3416>

   0x00002826 <+18>: movs r5, #2

   0x00002828 <+20>: strd r7, r8, [sp, #12]

...

(daisy-gdb) run

Starting program: /bin/grep shebang /bin/ls

qemu: Unsupported syscall: 26

#!/usr/bin/coreutils --coreutils-prog-shebang=ls

qemu: Unsupported syscall: 26

During startup program exited normally.

(daisy-gdb) quit

Note in the example above that, like "regular" gdb when given --args, you can pass the arguments for the program being debugged to the gdb wrapper script just by adding them to the command line after the name of the program being debugged (except that --args isn't needed). 

The commands below show how to copy your incrementally-compiled unit test binary and source file(s) to the appropriate sysroot and then start gdb with that binary (using the correct libraries, etc).

(cr) $ cd /build/lumpy/tmp/portage
(cr) $ mkdir shill-test
(cr) $ cd shill-test
(cr) $ cp <path-to-binary>/shill_unittest .
(cr) $ cp <path-to-src>/shill_unittest.cc .
(cr) $ gdb-lumpy
(gdb-lumpy) directory /tmp/portage/shill-test          # Tell gdb to add /tmp/portage/shill-test to the paths it searches for source files
(gdb-lumpy) file ./shill_unittest

If gdb is still looking for the source file in the wrong directory path, you can use 'set substitute-path <from> <to>' inside gdb to help it find the right path (inside your sysroot) for searching for source files.

---

---

Remote Debugging

---

Setting up remote debugging by hand.

If you want to manually run through all the steps necessary to set up your system for remote debugging and start the debugger, see Remote Debugging in Chromium OS.

---

Automated remote debugging using gdb-${BOARD} script (gdb-lumpy, gdb-daisy, gdb-parrot, etc).

gdb-${BOARD} is a script that automates many of the steps necessary for setting up remote debugging with gdb.  It should already be installed in your chroot.  If you do not have the script, update your repository to get the latest changes, then re-build your packages:

repo sync

./build_packages --board=...


This should install gdb_remote in the /usr/bin directory inside the chroot.  The gdb-${BOARD} script takes several options.  The most important ones are mentioned below.

"--gdb_args" ("-g") are arguments to be passed to gdb itself (rather than to the program gdb is debugging).  If multiple arguments are passed, each argument requires a separate -g flag. 

E.g gdb-lumpy --remote=123.45.67.765  -g "-core=/tmp/core" -g "-directory=/tmp/source"

"--remote" is the ip_address or name for your chromebook, if you are doing remote debugging.  If you omit this argument, the assumption is you are doing local debugging in the sysroot on your desktop (see section above).  if you are debugging in the VM, then you need to specify either ':vm:' or 'localhost:9222'.

"--pid" is the pid of a running process on the remote device to which you want gdb/gdbserver to attach.

"--attach" is the name of the running process on the remote device to which you want gdb/gdbserver to attach.  If you want to attach to the Chrome browser itself, there are three special names you can use:  'browser' will attach to the main browser process; 'gpu-process' will attach to the GPU process; and 'renderer' will attach to the renderer process if there is only one.  If there is more than one renderer process --attach=renderer will return a list of the renderer pids and stop.

To have gdb/gdbserver start and attach to a new (not already running) binary, give the name of the binary, followed by any arguments for the binary, at the end of the command line:

$ gdb-daisy --remote=123.45.67.809 /bin/grep "test" /tmp/myfile

When doing remote debugging you *must* use the --pid or the --attach option, or specify the name of a new binary to start.  You cannot start a remote debugging session without having specified the program to debug in one of these three ways.

When you invoke gdb-${BOARD} --remote=..., it will connect to the notebook or VM (automatically setting up port-forwarding on the VM), make sure the port is entered into the iptables, and start up gdbserver, using the correct port and binary, either attaching to the binary (if a remote pid or name was specified) or starting up the binary.  It will also start the appropriate version of gdb (for whichever type of board you are debugging) on your desktop and connect the gdb on your desktop to the gdbserver on the remote device.

---

Examples of debugging using the gdb-${BOARD} script.

Below are three examples of using the board-specific gdb wrapper scripts to start up debugging sessions.  The first two examples show connecting to a remote chromebook.  The first one automatically finds the browser's running GPU process, attaches gdbserver to the running process, starts gdb on the desktop, and connects the local gdb to gdbserver.  It also shows the user running  the 'bt' (backtrace) command after gdb comes up.  The second example shows the user specifying the pid of a process on the chromebook.  Again the script attaches gdbserver to the process, starts gdb on the desktop, and connects the two.  The third example shows the user connecting to the main browser process in ChromeOS running in a VM on the user's desktop.    For debugging the VM, you can use either

--remote=:vm: or --remote=localhost:9222  (":vm:" gets translated into "localhost:9222").

Example 1:


$ gdb-lumpy --remote=123.45.67.809 --attach=gpu-process

14:50:07: INFO: RunCommand: ping -c 1 -w 20 123.45.67.809
14:50:09: INFO: RunCommand: file /build/lumpy/opt/google/chrome/chrome
14:50:10: INFO: RunCommand: x86_64-cros-linux-gnu-gdb --quiet '--eval-command=set sysroot /build/lumpy' '--eval-command=set solib-absolute-prefix /build/lumpy' '--eval-command=set solib-search-path /build/lumpy' '--eval-command=set debug-file-directory /build/lumpy/usr/lib/debug' '--eval-command=set prompt (lumpy-gdb) ' '--eval-command=file /build/lumpy/opt/google/chrome/chrome' '--eval-command=target remote localhost:38080'
Reading symbols from /build/lumpy/opt/google/chrome/chrome...Reading symbols from/build/lumpy/usr/lib/debug/opt/google/chrome/chrome.debug...done.
(lumpy-gdb) bt 
#0  0x00007f301fad56ad in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f3020d5787c in g_main_context_poll (priority=2147483647, n_fds=3,   fds=0xdce10719840, timeout=-1, context=0xdce1070ddc0) at gmain.c:3584 
#2  g_main_context_iterate (context=context@entry=0xdce1070ddc0,block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)  at gmain.c:3285
#3  0x00007f3020d5798c in g_main_context_iteration (context=0xdce1070ddc0may_block=1) at gmain.c:3351
#4  0x00007f30226a4c1a in base::MessagePumpGlib::Run (this=0xdce10718580, delegate=<optimized out>) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/base/message_loop/message_pump_glib.cc:309
#5  0x00007f30226666ef in base::RunLoop::Run (this=this@entry=0x7fff72271af0) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/base/run_loop.cc:55
#6  0x00007f302264e165 in base::MessageLoop::Run (this=this@entry=0x7fff72271db0) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/base/message_loop/message_loop.cc:307
#7  0x00007f30266bc847 in content::GpuMain (parameters=...) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/content/gpu/gpu_main.cc:365
#8  0x00007f30225cedee in content::RunNamedProcessTypeMain (process_type=..., main_function_params=..., delegate=0x7fff72272380 at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/content/app/content_main_runner.cc:385
#9  0x00007f30225cef3a in content::ContentMainRunnerImpl::Run (this=0xdce106fef50) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/content/app/content_main_runner.cc:763
#10 0x00007f30225cd551 in content::ContentMain (params=...) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/content/app/content_main.cc:19
#11 0x00007f3021fef02a in ChromeMain (argc=21, argv=0x7fff722724b8) at ../../../../../chromeos-cache/distfiles/target/chrome-src-internal/src/chrome/app/chrome_main.cc:66
#12 0x00007f301fa0bf40 in __libc_start_main (main=0x7f3021fee760 <main(int, char const**)>, argc=21, argv=0x7fff722724b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,stack_end=0x7fff722724a8) at libc-start.c:292
#13 0x00007f3021feee95 in _start ()
(lumpy-gdb)

Example 2:


$ gdb-daisy --pid=626 --remote=123.45.98.765
14:50:07: INFO: RunCommand: ping -c 1 -w 20 123.45.98.765
14:50:09: INFO: RunCommand: file /build/daisy/usr/sbin/cryptohomed
14:50:10: INFO: RunCommand: armv7a-cros-linux-gnueabi-gdb --quiet '--eval-command=set sysroot /build/daisy' '--eval-command=set solib-absolute-prefix /build/daisy' '--eval-command=set solib-search-path /build/daisy' '--eval-command=set debug-file-directory /build/daisy/usr/lib/debug' '--eval-command=set prompt (daisy-gdb) ' '--eval-command=file /build/daisy/usr/sbin/cryptohomed' '--eval-command=target remote localhost:38080'
Reading symbols from /build/daisy/usr/sbin/cryptohomed...Reading symbols from/build/daisy/usr/lib/debug/usr/bin/cryptohomed.debug...done.
(daisy-gdb)

Example 3:


$ gdb-lumpy --remote=:vm: --attach=browser
15:18:28: INFO: RunCommand: ping -c 1 -w 20 localhost
15:18:31: INFO: RunCommand: file /build/lumpy/opt/google/chrome/chrome
15:18:33: INFO: RunCommand: x86_64-cros-linux-gnu-gdb --quiet '--eval-command=setsysroot /build/lumpy' '--eval-command=set solib-absolute-prefix /build/lumpy' '--eval-command=set solib-search-path /build/lumpy' '--eval-command=set debug-file-directory /build/lumpy/usr/lib/debug' '--eval-command=set prompt (lumpy-gdb) ' '--eval-command=file /build/lumpy/opt/google/chrome/chrome' '--eval-command=target remote localhost:48062'
Reading symbols from /build/lumpy/opt/google/chrome/chrome...Reading symbols from /build/lumpy/usr/lib/debug/opt/google/chrome/chrome.debug...done.
done.
Remote debugging using localhost:48062
...
(lumpy-gdb)

If you find problems with the board-specific gdb scripts, please file a bug (crosbug.com) and add 'build-toolchain' as one of the labels in the bug.

---

---

Troubleshooting

---

I lost my developer tools on the stateful partition, can I get them back?

This happens sometimes because the security system likes to wipe out the stateful partition and a lot of developer tools are in /usr/local/bin. But all is not lost because there is a tool for updating the stateful partition from an image created by the auto-update part of the dev_server. Sadly, it is normally found in /usr/local so will have been lost too and you need to copy it over manually. This works for me:

$ cd /tmp

$ scp me@myworkstation:/path/to/chromiumos/chroot/build/x86-whatever/usr/bin/stateful_update .

$ sudo sh stateful_update

$ sudo reboot

Note you can clobber the stateful partition (remove user accounts etc and force OOBE) as part of this process by using a flag:

$ cd /tmp

$ scp me@myworkstation:/path/to/chromiumos/chroot/build/x86-whatever/usr/bin/stateful_update .

$ sudo sh stateful_update --stateful_change=clean

$ sudo reboot

---

---

Running Tests

Many of the automated tests that are part of the Chromium OS project run using the autotest framework. See Autotest User Documentation for details.

Various quick links:

Creating a new test

---

Running Smoke Suite On a VM Image

---

Seeing which tests are implemented by an ebuild

---

Getting an image that has been modified for test - This doesn't work if you want to create a recovery image and boot a Cr48 with it, see below

If you wish to produce a VM image instead, you should omit the --test flag to build_image and let ./image_to_vm.sh produce the test image:

./image_to_vm.sh --board=${BOARD} --test_image 

Note: this difference between `cros flash` and ./image_to_vm.sh arises because ./image_to_vm.sh does not yet support the --image_name flag and by default looks for chromiumos_image.bin.  We expect this to change in the future.

Note that creating a test image will change the root password of the image to test0000.  The --test_image flag causes the image_to_xxx commands to make a copy of your chromiumos_image.bin file called chromiumos_test_image.bin (if that file doesn't already exist), modify that image for test, and use the test image as the source of the command.

SIDE NOTES:

• If the directory already has a file called chromiumos_test_image.bin, that file is reused. That means that calling both cros flash and image_to_vm.sh doesn't waste time by creating the test image twice.

---

Creating a recovery image that has been modified for test

After building a test image using ./build_image test as described above, you may wish to encapsulate it within a recovery image:

./mod_image_for_recovery.sh \

    --board=${BOARD} \

    --nominimize_image \

    --image ~/trunk/src/build/images/${BOARD}/latest/chromiumos_test_image.bin \
    --to ~/trunk/src/build/images/${BOARD}/latest/recovery_test_image.bin

If desired, you may specify a custom kernel with --kernel_image ${RECOVERY_KERNEL}.

You can write this recovery image out to the USB device like so:

cros flash usb:// ~/trunk/src/build/images/${BOARD}/latest/recovery_test_image.bin

Note that there are some downsides to this approach which you should keep in mind.

• Your USB image will be a recovery mode/test image, but the ordinary image in your directory will be a non-test image.
• If you use devserver, this will serve the non-test image not the test-image.
• This means a machine installed with a test-enabled USB image will update to a non-test-enabled one.

---

---

Additional information

Toolchain Compilers

At any given time in the chroot, to see what cross-compiler version is the current default one, do:

• For ARM: armv7a-cros-linux-gnueabi-gcc -v
• For x86: i686-pc-linux-gnu-gcc -v
• For amd64: x86_64-cros-linux-gnu-gcc -v

Developer mode

It can be confusing to figure out what "developer mode" means: Some parts of this document talk about switching your hardware to developer mode, while other parts talk about entering developer mode by running an image built with the --withdev flag. For the most part, the two actions enable the same set of things (like the shell command in crosh). Thus, you can think of either action as enabling developer mode.

To provide some details:

• Putting your hardware into developer mode makes /bin/cros_boot_mode print developer. It also makes the file /mnt/stateful_partition/.developer_mode appear.
• Building with the --withdev flag creates a file called /root/.dev_mode (in addition to adding a whole bunch of useful developer tools to your stateful partition). The presence of the /root/.dev_mode file enables developer mode and also prevents wiping of the stateful partition when you switch your hardware between developer mode and release mode.

---

Attribution requirements

When you produce a Chromium OS image, you need to fulfill various attribution requirements of third party licenses. Currently, the images generated by a build don't do this for you automatically. You must modify ~/chromium/src/chrome/browser/resources/chromeos/about_os_credits.html.

---

Documentation on this site

You now understand the basics of building, running, modifying, and testing Chromium OS, but you've still got a lot to learn. Here are links to a few other pages on the chromium.org site that you are likely to find helpful (somewhat ordered by relevance):

• The Tips And Tricks for Chromium OS Developers page has a lot of useful information to help you optimize your workflow.
• If you haven't read the page about the devserver already, read it now.
• Learn about the Chromium OS directory structure.
• The Chromium OS developer FAQ
• Chromium OS Portage Build FAQ is useful for portage-specific questions.
•  If you have questions about the --noenable_rootfs_verification option, you might find answers on this thread on chromium-os-dev.
• Running Smoke Suite on a VM Image has good information about how to get up and running with autotest tests and VMs.
• Debugging Tips contains information about how to debug the Chromium browser on your Chromium OS device.
• Working on a Branch has tips for working on branches.
  
• Git server-side information talks about adding new git repositories.
• The Portage Package Upgrade Process documents how our Portage packages can be upgraded when they fall out of date with respect to upstream.
• The Chromium OS Sandboxing page describes the mechanisms and tools used to sandbox (or otherwise restrict the privileges of) Chromium OS system services.
  
• The Go in Chromium OS page provides information on using Go in Chromium OS, including recommendations for project organization, importing and managing third party packages, and writing ebuilds.

---

External documentation

Below are a few links to external sites that you might also find helpful (somewhat ordered by relevance):

• Definitely look at the Chromium OS dev group to see what people are talking about.
• Check out the Chromium OS bug tracker to report bugs, look for known issues, or look for things to fix.
• Get an idea of what's actively happening by looking at the Chromium Gerrit site. (Note that this is no longer the same site that Chromium uses)
• Browse the source code on the Chromium OS gitweb.
• Check the current status of the builds by looking at the Chromium OS build waterfall.
• Check out the #chromium-os channel on freenode.net (this is the IRC channel that Chromium OS developers like to hang out on).
• If you're learning git, check out Git for Computer Scientists, Git Magic, or the Git Manual.
• Gentoo has several useful documentation resources
• Gentoo Development Guide is a useful resource for Portage, which is the underlying packaging system we use.
• Gentoo Embedded Handbook 
• Gentoo Cross Development Guide
• Gentoo Wiki on Cross-Compiling
• Gentoo Package Manager Specification 
• The repo user docs might help you if you're curious about repo.
• The repo-discuss group is a good place to talk about repo.