Developing PNaCl

Quick links

Other sites

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.

Native Client‎ > ‎PNaCl‎ > ‎

The following sections are for developers making changes to the PNaCl tools.

You will need to check out the entire native client project with svn or with git. Once you have the native client project checked out, you can check out the PNaCl sources.

Update to the latest version of Native Client / PNaCl by running:

gclient sync

Pre-built binaries for PNaCl are located in one of these directories, depending upon your architecture and operating system.

toolchain/pnacl_linux_x86/newlib/

toolchain/pnacl_mac_x86/newlib/

toolchain/pnacl_win_x86/newlib/

You must build and install the SDK libraries before using the toolchain outside of SCons:

pnacl/build.sh sdk newlib

Sanity Check: Compiling a test program

At this point, you probably want to test if the pre-built toolchain works. Let's start by creating a "hello world" program:

#include <stdio.h>

int main(int argc, const char *argv[]) {
printf("Hello World!\n");

return 0;

}

Add the PNaCl toolchain to your path for convenience:

PATH=${PATH}:toolchain/pnacl_linux_x86/newlib/bin # Adjust the name for your system

Now compile hello.c:

pnacl-clang hello.c -o hello.pexe

(NOTE: If you get an error about missing crt1.o or -lnacl, you must install the SDK. See the installation instructions in the previous section.)

You have now produced a "pexe", a PNaCl bitcode application. To run this pexe, it must be first translated to native code. Let's translate it to X86-32, X86-64 and ARM.

pnacl-translate -arch x86-32 hello.pexe -o hello_x86_32.nexe

pnacl-translate -arch x86-64 hello.pexe -o hello_x86_64.nexe

pnacl-translate -arch arm hello.pexe -o hello_arm.nexe

Now you have 3 native executables, one for each architecture. To run them inside of sel_ldr, use the helper script, native_client/run.py:

./run.py hello_x86_32.nexe

./run.py hello_x86_64.nexe

./run.py hello_arm.nexe # By default it will look for qemu to run the ARM nexe.

Testing with the pre-built toolchain with SCons

To use the PNaCl toolchain with the native client scons tests, just specify "bitcode=1" on the command-line.

Some sample command lines:

./scons platform=arm bitcode=1 # build everything

./scons platform=arm bitcode=1 smoke_tests # run smoke_tests

./scons platform=x86-32 bitcode=1 # build everything

./scons platform=x86-32 bitcode=1 smoke_tests # run smoke_tests

./scons platform=x86-64 bitcode=1 # build everything

./scons platform=x86-64 bitcode=1 smoke_tests # run smoke_tests

The scons tests are more conveniently wrapped in a testing script:

pnacl/test.sh test-x86-32 # Runs the smoke tests as above, plus nonpexe_tests

pnacl/test.sh test-x86-64

pnacl/test.sh test-arm

pnacl/test.sh test-x86-32-sbtc # Runs the same tests, but with the sandboxed translator

pnacl/test.sh test-x86-64-sbtc

pnacl/test.sh test-arm-sbtc

Other Compiler Tests

LLVM regression tests

These tests can be run directly by running 'make check' in the build directory (pnac/build/llvm_x86_64). There are currently a few known failures which are not marked as XFAIL in the usual LLVM way, so instead the regression tests can also be run in the following way, which will ignore the known failures.

pnacl/scripts/llvm-test.sh llvm-regression

The LLVM regression tests run quickly and should be run before every LLVM commit, along with the scons tests.

The following tests take longer and are generally not run by the developers on every commit, but they should be run for large changes, and are run on the toolchain testing bots on every commit.

GCC Torture tests

To run the gcc torture test suite against pnacl, do the following:

tools/toolchain_tester/torture_test.sh install-tests # Only needs to be run once

tools/toolchain_tester/torture_test.sh pnacl-x86-64-torture --concurrency=12

pnacl-x86-32-torture and pnacl-arm-torture are other targets. Other options are documented in torture_test.sh and tools/toolchain_tester/toolchain_tester.py

LLVM Test Suite

To run the LLVM testsuite against pnacl, do

pnacl/scripts/llvm-test.sh all x86-64

This is equivalent to

pnacl/scripts/llvm-test.sh testsuite-clean

pnacl/scripts/llvm-test.sh testsuite-configure

pnacl/scripts/llvm-test.sh testsuite-run x86-64

pnacl/scripts/llvm-test.sh testsuite-report x86-64

The testsuite needs to be cleaned and re-configured each time you switch architectures, but the results files persist after cleaning, so testsuite-report will give the result from the last run for that architecture. Because there is no good way to filter the set of tests that are actually run and there are a lot of known failures (e.g. tests that have not been ported to use newlib), this testsuite takes a long time to run.

Spec2k

If you have access to the spec2k benchmark suite you can use the harness in tests/spec2k/ to run some more extensive tests.

If SPEC_DIR is the directory containing the benchmark code/data, run:

pnacl/test.sh test-spec <SPEC_DIR> SetupPnaclX8664Opt

test.sh is a convenience script and clobbers old results, etc. The actual steps are:

pushd tests/spec2k

./run_all.sh CleanBenchmarks

./run_all.sh PopulateFromSpecHarness <SPEC_DIR>

./run_all.sh TimedBuildAndRunBenchmarks SETUP

popd

where SETUP is for example "SetupPnaclArmOpt". See "run_all.sh" for a full list of setups.

Toolchain Development

TL;DR for checking out PNaCl sources, building and testing

From the native_client/ directory:

pnacl/build.sh clean

pnacl/build.sh sync-sources # Pulls the PNaCl sources

pnacl/build.sh build-all # Builds and installs PNaCl toolchain

pnacl/build.sh translator-all # Builds the sandboxed translator

If everything built correctly, run the scons tests with:

Important Scripts

Most tasks related to the toolchains are focused in two scripts.

the driver scripts in the directory: pnacl/driver/
the toolchain builder and helper script: pnacl/build.sh

pnacl/build.sh

This script checks out and builds the complete PNaCl toolchain. Important invocations:

sync-sources: checks out and/or updates the toolchain sources
build-all: builds the toolchain incrementally
all: alias for sync-sources + build-all
translator-all: builds the sandboxed translator nexes (not included in 'build-all' or 'all')
clean: cleans the build files
driver: reinstalls a locally modified driver script
help: concise help
help-full: verbose help

pnacl/test.sh

This script runs toolchain tests. It deletes old testing artifacts before running. The main invocation is test-all.

pnacl/driver/*

This directory contains the compiler drivers, which are in Python. If you change them, run "pnacl/build.sh driver" to install them.

Git Repositories

Most of the code required to build the PNaCl toolchain lives in git repositories hosted on git.chromium.org. The build.sh script will automatically check out and update these repositories. The checked-out code is placed in native_client/pnacl/git.

Relevant Directories

pnacl/git	holds all the git check outs
toolchain/pnacl_${os}_${arch}	root of PNaCl toolchain
pnacl/build/	holds build directories for the pnacl toolchain
pnacl/log/	olds the logs of the builds

Process for making changes to the PNaCl toolchain

If you've made changes to PNaCl toolchain, either to the driver or git repositories, you can rebuild PNaCl by doing:

pnacl/build.sh all # Incrementally syncs, builds and reinstalls

pnacl/test.sh test-all # Run the scons tests

How to send out reviews for git repos

To send out reviews for the git repos, you must have depot tools. Code reviews are sent using the git cl script in depot_tools.

If you are going to commit, you also need an account on gerrit.chromium.org and an SSH key setup for it.

The first time you set up code review you'll need to let git cl do some setup:

% cd pnacl/git/llvm
% git cl config

Just press Enter at the prompts.

To do local development, create a new branch:

% git checkout -b mybranch origin/master

The last argument to git checkout (the start point) is important because if your local branch does not track the remote master branch, then git cl push will not work)

Edit your code, and commit as many times as you want:

% git commit -a -m “comment describing changes”

Then use git-cl to upload to the code review site:

% git cl upload

You'll need to authenticate to codereview.chromium.org also. Most likely this will be your ldap email. This will upload the review, and give you an issue number, e.g. http://codereview.chromium.org/1557002. Go to the issue webpage, and edit the change summary, add reviewers, then hit send mail. Command line options from gcl such as reviewers ("-r") and send mail ("--send_mail") are also available with git cl.

Git cl associates each branch with a review issue ID. So you if you type git cl upload again from the same branch, it will upload a new patch set to the same issue.

When you get the LGTM, you can push the changes.

% git cl push

When you push, you will need to have your gerrit SSH key to authenticate to the gerrit git server.

If there have been changes on the master branch since you forked your local branch, git cl may complain, in which case you should rebase it against the new HEAD before you push:

% git fetch origin

% git rebase origin/master

Update stable revisions in the pnacl DEPS, test, review and submit

At this point the official toolchain builder is still not incorporating your changes as it builds off of specific commit IDs in the external repo.

To fix this modify native_client/pnacl/DEPS to point at your new commit ID which look like

"pnacl_llvm_rev": "efa7b3d5c85f0242787eb091dadedc2727215df4",

You can then make a standard native_client change list and test the CL with trybots (CLs affecting pnacl/ will end up selecting pnacl trybots). Upload, trybot, get an LGTM, then commit. You can also test locally by cleaning the toolchain directory, rebuilding, and testing as described earlier.

Change the native client TOOL_REVISIONS, test, review and submit

Finally, to allow users to download the new revision built by the toolchain bulder, look in the native_client/TOOL_REVISIONS file for

PNACL_VERSION=9476

and update this to an svn revision number that is new enough to include your build.sh revision change. Now create a CL and test it with some try jobs (e.g., gcl try my_cl_name). NOTE: You will need to wait for a toolchain to be built (see the waterfall) before this change actually works (downloads a toolchain tarball).