AddressSanitizer (ASan)

AddressSanitizer (ASan) is a fast memory error detector based on compiler instrumentation (LLVM).
It is fully usable for Chrome on Linux and Mac. There's a mostly-functional Windows port in progress too, and you can using SyzyASan.
Additional info on the tool itself is available at http://clang.llvm.org/docs/AddressSanitizer.html.

For the memory leak detector built into ASan, see LeakSanitizer. If you want to debug memory leaks, please refer to the instructions on that page instead.

Buildbots and trybots

The Chromium Memory waterfall (not to be confused with the Memory FYI waterfall) contains buildbots running Chromium tests under ASan on Linux (Linux ASan/LSan bots for the regular Linux build, Linux Chromium OS ASan for the chromeos=1 build running on Linux), OS X (both 32 and 64 bits), Chromium OS (x86 and amd64 builds running inside VMs). Linux and Linux Chromium OS bots run with --no-sandbox, but there's an extra Linux bot that enables the sandbox (but disables LeakSanitizer).
The trybots running Chromium tests on Linux and OSX are: linux_asan (everything except browser_tests and content_browsertests), linux_browser_asan (browser_tests and content_browsertests), mac_asan (many tests including browser_tests and content_browsertests), linux_chromeos_asan (the chromeos=1 build running on a Linux machine, many tests including browser_tests and content_browsertests).

Blink bots: WebKit Linux ASAN buildbot and linux_layout_asan trybot.

Pre-built Chrome binaries

You can grab fresh Chrome binaries for Linux built with ASan here

Build tests with ASan

If you're on mac or linux64, building with ASan is easy:

Make sure you've run tools/clang/scripts/update.sh (see https://code.google.com/p/chromium/wiki/Clang for details).

Linux64/Mac build

GYP_DEFINES='asan=1' gclient runhooks
ninja -C out/Release base_unittests
Start with base_unittests and verify your build is working (see below).  Then you can ninja ... chrome or ninja ... browser_tests.

Mac 64-bit build

    GYP_DEFINES='asan=1 target_arch=x64 host_arch=x64gclient runhooks
    ninja -C out/Release base_unittests

On both Linux64 and Mac, building with ninja instead of make should just work. Make sure to do a release build.

Adding use_allocator=none to GYP_DEFINES also appears to be necessary as of 19/06/2014, but perhaps that's an unintended regression.

Windows 32-bit build

DUCT TAPE ALERT! EXPERIMENTAL!
ASan for Windows is not yet production quality.  It is based on Clang for Windows and inherits the Clang/Win list of unsupported stuff.  You can try it, but it might just be broken for you.
If you are assigned an ASan/Win bug, you might not be able to reproduce the report locally.  In this case, you have to look at the stack on the bug, and you have to check in a speculative fix and rely on the infra bots to tell you if your fix worked.

If you're feeling lucky, you can try using ASan/Win locally by running the steps below:
    # You should have CMake installed and added to your PATH
    set GYP_DEFINES=asan=1 disable_nacl=1
    gclient runhooks
    ninja -C out\Release base_unittests

Currently, ASan doesn't support Windows sandboxing, so make sure to pass --no-sandbox when you run chrome.exe.

Goma build

ASan builds should work seamlessly with Goma (except for Windows); just add use_goma=1 to your GYP_DEFINES.

Build options

If you want your stack traces to be precise, you will have to disable inlining: 
GYP_DEFINES='... release_extra_cflags="-g -O1 -fno-inline-functions -fno-inline" ' gclient runhooks

Verify the ASan tool works

ATTENTION (Linux only): These instructions are for running ASan in a way that is compatible with the sandbox. However, this is not compatible with LeakSanitizer. If you want to debug memory leaks, please use the instructions on the LeakSanitizer page instead.

Now, check that the tool works. Run the following: 

export LLVM_SYMBOLIZER_PATH=third_party/llvm-build/Release+Asserts/bin/llvm-symbolizer
# strict_memcmp=0 - http://crbug.com/178677
# replace_intrin=0 - http://crbug.com/162461 (Mac only)
export ASAN_OPTIONS="strict_memcmp=0 replace_intrin=0"
out/Release/base_unittests --gtest_filter=ToolsSanityTest.DISABLED_AddressSanitizerLocalOOBCrashTest --gtest_also_run_disabled_tests 2>&1 | tools/valgrind/asan/asan_symbolize.py

The test will crash with the following error report:
==26552== ERROR: AddressSanitizer stack-buffer-overflow on address 0x7fff338adb14 at pc 0xac20a7 bp 0x7fff338adad0 sp 0x7fff338adac8
WRITE of size 4 at 0x7fff338adb14 thread T0
    #0 0xac20a7 in base::ToolsSanityTest_DISABLED_AddressSanitizerLocalOOBCrashTest_Test::TestBody() ???:0
    #1 0xcddbd6 in testing::Test::Run() testing/gtest/src/gtest.cc:2161
    #2 0xcdf63b in testing::TestInfo::Run() testing/gtest/src/gtest.cc:2338
...
Address 0x7fff338adb14 is located at offset 52 in frame <base::ToolsSanityTest_DISABLED_AddressSanitizerLocalOOBCrashTest_Test::TestBody()> of T0's stack:
  This frame has 2 object(s):
    [32, 52) 'array'
    [96, 104) 'access'
==26552== ABORTING

Congrats, you have a working ASan build! :-)

Run chrome under ASan
And finally, have fun with the out/Release/chrome binary. The filter script tools/valgrind/asan/asan_symbolize.py should be used to symbolize the output.
(Note that asan_symbolize.py is absolutely necessary if you need the symbols - there is no built-in symbolizer for ASan in Chrome).

ASan should perfectly work with Chrome's sandbox. You should only need to run with --no-sandbox on Linux if you're debugging ASan.
Note: you have to disable the sandbox on Windows until it is supported.

You may need to run with --disable-gpu on Linux with NVIDIA driver older than 295.20.

You will likely need to define environment variable G_SLICE=always-malloc to avoid crashes inside gtk.
NSS_DISABLE_ARENA_FREE_LIST=1 and NSS_DISABLE_UNLOAD=1 are required as well.

When filing a bug found by AddressSanitizer, please add a label Stability-AddressSanitizer.


ASan runtime options

ASan's behavior can be changed by exporting the ASAN_OPTIONS env var. Some of the useful options are listed on this page, others can be obtained from running an ASanified binary with ASAN_OPTIONS=help=1.
Note that Chromium sets its own defaults for some options, so the default behavior may be different from that observed in other projects.
See base/debug/sanitizer_options.cc for more details.

NaCl support under ASan

On Linux (and soon on Mac) you can build and run Chromium with NaCl under ASan. Untrusted code (nexe) itself is not instrumented with ASan in this mode, but everything else is.

To do this, remove disable_nacl=1 from GYP_DEFINES, and define NACL_DANGEROUS_SKIP_QUALIFICATION_TEST=1 in your environment at run time.

Pipe chromium output (stderr) through tools/valgrind/asan/asan_symbolize.py `pwd`/ to get function names and line numbers in ASan reports.
If you're seeing crashes within nacl_helper_bootstrap, try deleting out/Release/nacl_helper.


Building on iOS

It's possible to build and run Chrome tests for iOS simulator (which are x86 binaries essentially) under ASan. Note that you'll need a Chrome iOS checkout for that. It isn't currently possible to build iOS binaries targeting ARM.

Fix chromium.gyp_env to use ASan.

$ gyp_env=`ls ios_*/build/chromium.gyp_env`
$ cp $gyp_env $gyp_env.orig
$ cat $gyp_env.orig | sed "s/\('GYP_DEFINES.*\)' \+/\1 asan=1 clang_use_chrome_plugins=0' +/" > $gyp_env

Then update Clang and prepare the build env:
./tools/clang/scripts/update.sh
gclient runhooks
export ASAN_BIN=`pwd`/third_party/llvm-build/Release+Asserts/bin/ export CC=$ASAN_BIN/clang export CXX=$ASAN_BIN/clang++

    Now build the test binary and run it:

xcodebuild -project base/base.xcodeproj -target base_unittests -sdk iphonesimulator
xcodebuild/ninja-iossim/Debug/iossim xcodebuild/Debug-iphonesimulator/base_unittests.app/ \
    --gtest_filter=ToolsSanityTest.DISABLED_AddressSanitizerLocalOOBCrashTest --gtest_also_run_disabled_tests 2>&1 | tools/valgrind/asan/asan_symbolize.py

You'll see the same report as shown above (see the "Verify the ASan tool works" section), with a number of iOS-specific frames.

Building on Android

Follow AndroidBuildInstructions with minor changes:

gclient sync --nohooks
. build/android/envsetup.sh
GYP_DEFINES="OS=android asan=1 component=shared_library" gclient runhooks

component=shared_library is required to link shared libstlport.so instead of a static one, which allows AddressSanitizer to interpose operators new and delete. Omitting this flag will get you broken allocation and deallocation stacks, but things will generally work.

Running ASan applications on Android requires additional device setup. Chromium testing scripts take care of this, so testing works as expected:
build/android/test_runner.py instrumentation --test-apk ContentShellTest --test_data content:content/test/data/android/device_files -v -v -v --tool=asan --release

To run stuff without Chromium testing script (ex. ContentShell.apk, or any third party apk or binary), device setup is needed:
./tools/android/asan/asan_device_setup.sh
# wait a few seconds for the device to reload
It only needs to be run once per device. It is safe to run it multiple times.
When this is done, the device will run ASan apks as well as normal apks without any further setup.

To run command-line tools (i.e. binaries), prefix them with asanwrapper:
adb shell /system/bin/asanwrapper /path/to/binary

Building with v8_target_arch=arm

This is needed to detect addressability bugs in the ARM code emitted by V8 and running on an instrumented ARM emulator in a 32-bit x86 Linux Chromium. You probably don't want this.
See http://crbug.com/324207 for some context.

First, you need to install the 32-bit chroot environment using the build/install-chroot.sh script (as described in https://code.google.com/p/chromium/wiki/LinuxBuild32On64)
Second, install the build deps:
precise32 build/install-build-deps.sh  # assuming your schroot wrapper is called 'precise32'

You'll need to make two symlinks to avoid linking errors:
sudo ln -s $CHROOT/usr/lib/i386-linux-gnu/libc_nonshared.a /usr/lib/i386-linux-gnu/libc_nonshared.a
sudo ln -s $CHROOT/usr/lib/i386-linux-gnu/libpthread_nonshared.a /usr/lib/i386-linux-gnu/libpthread_nonshared.a

Now configure and build your Chrome:
GYP_GENERATOR_FLAGS="output_dir=out_asan_chroot" GYP_DEFINES="asan=1 disable_nacl=1 v8_target_arch=arm sysroot=/var/lib/chroot/precise32bit/ chroot_cmd=precise32 host_arch=x86_64 target_arch=ia32" gclient runhooks
ninja -C out_asan_chroot/Release chrome

Note: disable_nacl=1 is needed for now.

Linux/Mac build using a custom Clang

If for some reason you need to build a 32-bit Chrome binary, you'll need to do some magic. The same command is needed to use a custom compiler binary:
cd src
ASAN_BIN=/path/to/clang/bin
export CC="$ASAN_BIN/clang"
export CXX="$ASAN_BIN/clang++"
GYP_DEFINES='asan=1 use_allocator=none clang_use_chrome_plugins=0' gclient runhooks
ninja -C out/Release base_unittests

AsanCoverage

AsanCoverage is a minimalistic code coverage implementation built into ASan. For general information see https://code.google.com/p/address-sanitizer/wiki/AsanCoverage
To use AsanCoverage in Chromium, add asan_coverage=1 to your regular ASan GYP_DEFINES. This gets you function level coverage. To get block level coverage, use asan_coverage=2.
Note that renderers and the GPU process will not generate one coverage file per process per module, as would normally be the case. Instead, there will be two .sancov.packed files which will have to be unpacked to regular .sancov files using sancov.py unpack. This is required to get around sandboxing restrictions.

Example usage:

$ GYP_DEFINES='asan=1 asan_coverage=1' gclient runhooks
$ ninja -C out/Release chrome
$ ASAN_OPTIONS="strict_memcmp=0 symbolize=false coverage=1" out/Release/chrome

Chrome must be terminated gracefully in order for coverage to work. Either close the browser, or SIGTERM the browser process. Do not do killall chrome or send SIGKILL.

$ kill <browser_process_pid>
$ ls
...
chrome.22575.sancov
gpu.6916123572022919124.sancov.packed
zygote.13651804083035800069.sancov.packed
...

The gpu.*.sancov.packed file contains coverage data for the GPU process, whereas the zygote.*.sancov.packed file contains coverage data for the renderers (but not the zygote process). Unpack them to regular .sancov files like so:

$ $LLVM/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py unpack *.sancov.packed
sancov.py: unpacking gpu.6916123572022919124.sancov.packed
sancov.py: extracting chrome.22610.sancov
sancov.py: unpacking zygote.13651804083035800069.sancov.packed
sancov.py: extracting libpdf.so.12.sancov
sancov.py: extracting chrome.12.sancov
sancov.py: extracting libpdf.so.10.sancov
sancov.py: extracting chrome.10.sancov

Now, e.g., to list the offsets of covered functions in the libpdf.so binary in renderer with pid 10:

$ $LLVM/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py print libpdf.so.10.sancov


Please refer to the ASan wiki for more information.

Comments