TODO: Needs an introduction that someone who's a bit more of a newbie can understand.
Good Gentoo documentation:
TODO: this section needs to be aimed more at portage / Chromium OS newbie level.
Portage is designed to compile packages from source and install them into a root directory. We take advantage of this to make a distinction between host and our target when building. For the host, the root directory where things get installed is /, for a given target board this is /build/<board_name>. You can think of each as a mini Linux system, although they start off with hardly any packages installed.
In order to cross-compile, you need some tools that are run on the host in order to build individual packages. When we set up the system, we install the hard-host-depends package to cover these. When additional host tools are needed we can update the package and re-merge it on the host, which will pull in any new tools that are needed -- the build_packages script does this automatically. Notice that since we are updating the host, we use the standard emerge command that installs packages at the root and uses the host's configuration:
We set up a new sysroot per target board that we want to cross compile for. First we make sure that the desired toolchain is installed on the host and set up --sysroot wrappers for the toolchain binaries that set the sysroot on compile. We use the toolchain headers and libraries to bootstrap the board target sysroot in /build/<board_name>. When we set up the board, we can optionally set up a Portage overlay for that individual board. This allows us to customize or add packages on a per-board basis. The board overlay can also include a make.conf that overrides some of our defaults such as compiler CFLAGS.
When cross-compiling, you also need packages that correspond to the target, such as header files and libraries. Our strategy is to install them in the target's root using the --root-deps option. These are only used for compilation and will not be placed into images that are built via build_image.
Once a board is set up, we can start building packages for the board target. We use the board target specific version of emerge to do this.
Building a full bootable image is a multi-step process. For a given target, the first step is to build binary packages for all packages needed by Chromium OS using
We've set a portage option to build binary packages as a side-effect of building anything from source. If you build all of the packages needed by the chromeos ebuild then we will have a binary package for each of these.
Once build_packages finishes, we'll run
We use the
SIDE NOTE: It turns out that for most developers, the above is a bit of a lie. Most developers don't actually build all binary packages from source when they do a build. By default, the build system "cheats" and downloads a binary package from the BINHOST if it can find one. See What does build_packages actually do.
Here are a few important directories:
One of the interesting aspects of the Portage build system is parameterized dependencies and build options. Each package can declare a list of USE flags that can be turned on and off. Turning them on and off can affect the set of packages that they depend on. For example, if a package supports creating optional perl bindings, then it can declare a perl USE flag. If "-perl" is specified then the package can omit creation of the bindings and drop its dependency on perl.
The easiest way to inspect dependencies is to use the --emptytree and --pretend options for emerge. These options tell you what packages would be installed if you were installing from scratch and you didn't have any of its dependencies yet. The --verbose option will also show the set of USE flags that can be turned on and off for the packages to hopefully trim dependencies:
To get an idea of the dependency tree and why some packages would be built do parallel_emerge --pretend --emptytree --tree like so:
If you wanted to look for busybox, you can:
An ebuild file is the recipe to build a package. Since it is written in shell script it is very flexible, but there is a recommended format. Start with one of our existing chromium ebuilds as an example. The ebuild should be named project-version.ebuild and it should live in a directory under an appropriate category. For chromiumos packages we've been using the
The board setup process creates an
If you are writing a new package for that does not exist upstream then you will want to upload the files for the package to the localmirror. See the "download mirror" section below.
When we download packages for use in Chromium OS, we only download packages that are present on the upstream gentoo mirror or on our localmirror. If your package is not present on either download mirror, then you need to upload it manually to localmirror.
Once you have uploaded your package to the mirror, you can test that it's working by running the following commands:
When you are working on a cros_workon package (one with a 9999 ebuild), you only need to modify the 9999 ebuild - the Commit Queue will auto-uprev the version of the ebuild by copying the 9999 ebuild to an ebuild with the new bumped revision.
But, when you are working on a "non cros_workon" package (i.e a package for whose ebuild doesn't inherit "cros_workon", chromiumos doesn't host a git tree, and autouprev isn't supported) you must manually increase the version number in the ebuild filename.
When upgrading to a new version of the package (i.e. grabbing a newer version from upstream), then increase the ebuild version number (foo-M.m.ebuild).
When changing the ebuild contents (i.e. applying a custom patch or fixing cross-compile issues), then increase the ebuild revision number (foo-M.m-rN.ebuild).
When uprevving ebuilds, just rename the symlink (by convention, named as
If updating packages versions (say from PV=0.0.1 to PV=0.1.0 above), rename the base file (i.e.,
There are a few things that are important to get a debug package:
See Using the dev server.
See the How to build a single package and install it without doing a full update section of the dev server page.
Mount the image from file or USB stick using:
Emerge into the mounted system:
When importing a package with emerge-<board_name>, you may get an error message about "masked packages". For instance, the response to
may contain the following:
To unmask a package, edit the KEYWORDS field of the ebuild directly and add "amd64 arm x86". If you see existing entries like "~amd64", simply remove the tilde (~).
How do I build a package without it deleting the build directory (eg, to see a kernel .config file)?
Within Chromium OS the portage repository is considered read-only. This means that if the portage package has a problem then we must create an overlay for it in chromiumos-overlay. A common reason for needing to do this is that the package will not cross compile (for ARM or x86). Here is how to do that:
Note: You must run
Sometimes when using
This tends to happen when files move between packages, or if the upstream Gentoo packages in your chroot become inconsistent.
Sometimes you have one package (let's call it "pkgsrc" for short) providing a file but want to change things so that a different package ("pkgdst") now provides that file. If you were to try to upgrade "pkgdst" and have it install the file but the old "pkgsrc" still installs that file, we get into a bad state.
To solve this, we leverage package blockers. At the simplest level, this is an ebuild dependency syntax that allows you to say "do not allow pkg A to be installed whilst pkg B is installed". But if you combine the blocker syntax with versions, portage is smart enough to detect that it needs to handle these blocking packages specially and do an automatic upgrade for you.
Let's use a real world example. The package sys-fs/e2fsprogs-libs used to install the libuuid library. But the upstream developers decided to move it to the sys-apps/util-linux package. Specifically, the e2fsprogs-libs-1.41.7 release was the last one to include the library while util-linux-2.16 was the first one to include it.
So in the util-linux-2.16 ebuild, we declare that at runtime, it cannot be installed simultaneously with e2fsprogs-libs-1.41.7 or older. Since util-linux does not need e2fsprogs-libs to build libuuid, we don't have to declare the blocker in DEPEND.
Since e2fsprogs-libs doesn't actually need libuuid itself in order to build, there is no dependency in that ebuild on a newer util-linux version. However, let's assume it did. That'd mean in the e2fsprogs-libs-1.41.8 ebuild, we would write:
This is enough information for portage to be able to automatically resolve this blocker for you. It will make sure that when upgrading util-linux and e2fsprogs-libs, it will first upgrade util-linux, then ignore file collisions that it hits with e2fsprogs-libs (since util-linux is taking ownership of those files), then upgrade e2fsprogs-libs. If there was a bug and e2fsprogs-libs still tried to install libuuid, portage would throw an error because util-linux now owns those files.
For example, you had tried to
To recover from mixing of upstream packages, you can:
To find out which package provides the offending file and unmerge it, try:
To deal with moving files between packages, you'll want to utilize blockers
Depending on the options you give, other packages may be emerged also (note that many of these options are the default):
A small but important detail is that normally
If you are using the minilayout then you will not have downloaded all the source for Chrome OS. So when an ebuild is emerged it may need to download some source. For example, see the ebuild in:
It has some SRC_URI lines describing where the source package can be downloaded from. The file used in case of tar may be something like:
This file is cached for you after the first download, in, for example,
If you want to try this out, type:
You should see it download some source, build it and install it into your target build in
This time you will probably see it download something like:
and then simply install the package as before. The C compiler will not be touched. Again the file is cached, this time in something like
A virtual package is used in portage when any of several different packages can perform the same function. A classic example is "virtual/editor". There are dozens of editors out there (including the most fabulous one of them all:
Depending on a virtual package is pretty easy: just add a dependency (either DEPEND or RDEPEND) on the virtual, like:
Depending on a virtual is very easy, but what about if you want to add a new virtual? First: create an ebuild inside the virtual category with a "-0" version like
It's easiest to look at two samples.
First, we'll look at virtual/editor from upstream portage. The important things to notice are that this ebuild doesn't do anything itself--it just specifies dependencies. Specifically, it says that any of the editors listed are OK by specifying this as the RDEPEND (note the
In this case the ebuild doesn't provide any particular way to choose which editor is chosen--it just specifies all of the options. As long as one of those options is installed,
A second example is Chromium OS's virtual/linux-sources ebuild. You can see the important bits here:
You can see that the kernel ebuild specifies a way to choose what implementation via
You should definitely pay attention to the fact that virtuals are centrally-managed. Said another way: if you need to add another implementation of a virtual (like that nifty new
There is one common (at least in Chromium OS) case where it's not be possible to have things centrally managed. This happens when you've got an implementation of a virtual that's in your private overlay. We'll imagine a virtual called
You can do this by overriding the virtual in your private overlay. Your overlay will have a copy of the
When portage searches for packages to install, it merges all of the packages in all the overlays. If the same ebuild (category/package-name) is found in multiple places, it comes down to comparing versions such that the highest version wins (and if the same version is found, then overlays will "whiteout" other overlays based on search order). But that's a lot to keep straight and is fairly fragile. If the ebuild in the main tree is revbumped to a version higher than is in an overlay, then bad things could start happening.
The policy we have in place for versioning of our virtuals is as follows:
The above description is for new-style virtuals. There's also old-style virtuals. You shouldn't ever add one of those, but you might still run into them.
Declaring an old-style virtual is easy, but inflexible. In the
The first element is the name of the virtual while the second element is the package that provides that virtual. This file format has no other options/extensions available to it.
Portage's dependency system can be tricky to understand, especially when it comes to what will cause a package to be rebuilt.
Important: package manager's dependencies are NOT like make dependencies. If package B depends on package A, it does NOT mean B is re-built when A is updated. It only means when B is pulled in, A is also pulled in.
Just to make sure we're on the same page, there are two types of dependencies:
Often times we will have both a DEPEND and an RDEPEND on a package. A good example of this is when we depend on a ".so" file. We need the shared library's header file at build time and it's ".so" file at runtime.
To think about how everything works, we'll pretend we have three ebuilds:
OK, so what happens when we make changes and rebuild things. Here is what happens:
The above is a little weird, but can be understood by remembering that portage is primarily concerned with speed and correctness. ...but portage is not super concerned with making sure you have the latest version of every last dependency (as long as you have some version it is happy). Specifically:
Note that the above examples use a static library and dynamic library as an example. Hopefully it makes sense that we can map other problems to the same concepts. For instance, if we had an ebuild that took a whole bunch of other build outputs and created a ".zip" file out of them, that would be just like the static library case.
People working on Chrome OS probably find portage's philosophy (that getting the newest version isn't important) a bit frustrating. If an engineer syncs down new source code and tries to build a new boot image, the engineer would hope that the image has all of the newest versions of all of the packages.
We'll take chromeos-bootimage as an example. This build wants to take the binary output of several other ebuilds (the BCT, the firmware, the flattened device tree, etc) and concatenate them together (with some extra processing) to make a single binary image. Technically, chromeos-bootimage should only DEPEND on all of the other components. However, that means that if a new version of the BCT is checked in and then we do a build, portage will not decide to re-build chromeos-bootimage. Ick.
As a hack, Chrome OS packages often say that they DEPEND and RDEPEND in cases like chromeos-bootimage. Now if you build all packages, portage will be sure to rebuild chromeos-bootimage if the BCT ever changes. This is an awkward way to do things but is the current workaround. The fabled ABIDEPEND feature of portage (doesn't exist yet) is supposed to fix this and make it so we don't need the hack anymore.