For Developers‎ > ‎How-Tos‎ > ‎Using Valgrind‎ > ‎

ThreadSanitizer (TSAN)

There is a Google-grown Valgrind tool called ThreadSanitizer (aka TSan), which finds data races (note: if you've been assigned a bug that says "ThreadSanitizer v2", there is a different page about the new tool: ThreadSanitizer (TSan) v. 2)
We have three ThreadSanitizer build bot slaves set up on the main waterfall for Windows, Linux and Mac.
Also, we have a ThreadSanitizer UI bot on the FYI waterfall.
If curious, see the list of Chromium races found by ThreadSanitizer.

One of the most scary races found by ThreadSanitizer is a data race on reference counter which caused dozens of other bugs.
Together, these bugs were the reason of chromium top crasher for several months!
It took less than an hour to detect the race with
ThreadSanitizer while it was completely overlooked for months of testing and debugging.

Running ThreadSanitizer on Chromium (Linux and Mac)

In order to use it, you'll need to get ThreadSanitizer binaries from SVN.

Then, make sure you have your local GYP flags overriden like you should do for Valgrind (see using-valgrind).

To run Chromium tests under ThreadSanitizer, simply add "--tool tsan" to your Valgrind command, e.g.:
sh tools/valgrind/chrome_tests.sh -t base --tool tsan

See the Thread Sanitizer home page for more info on the tool itself.
Please note that by default ThreadSanitizer runs on Chromium in conservative mode (--pure-happens-before=yes).
You can pass additional flags to
ThreadSanitizer by using --tool_flags="--some-tsan-flag"

Running ThreadSanitizer on Chromium (Windows)

Most of the Linux and Mac instructions are still valid but there are specifics.

To get the binaries from SVN please add the following line to your .gclient file instead of using Valgrind instructions:
  { "name"        : "src",
    "url"         : <...>,
    "custom_deps" : {
      <your old custom_deps here>
      "src/third_party/tsan": "http://src.chromium.org/chrome/trunk/deps/third_party/tsan",
  },

You can only run Debug builds under TSan on Windows.
They can be slow, so to speed them up the following GYP values are recommended:
{
  'variables': {
    'build_for_tool': 'tsan',
  },
}
or just
set GYP_DEFINES="build_for_tool=tsan"
gclient runhooks

Now you can run your Chromium test using (notice .bat instead of .sh):
tools\valgrind\chrome_tests.bat -t base --tool tsan

Suppressing data races

ThreadSanitizer suppression format is very similar to that of Memcheck, so you can create suppressions the same way as you do for Memcheck.
Good news: ThreadSanitizer also supports demangled suppressions!

The ThreadSanitizer suppression file are located at tools/valgrind/tsan/suppression.txt and tools/valgrind/tsan/suppression_{linux,mac}.txt

The ThreadSanitizer runner script generates suppressions:
WARNING: Possible data race during write of size 8 at 0x7FFEFFF7D8: {{{
   T0 (L{}):
    #0  base::Watchdog::~Watchdog base/threading/watchdog.cc:33
    #1  base::::WatchdogCounter::~WatchdogCounter base/threading/watchdog_unittest.cc:28
    #2  base::WatchdogTest_AlarmPriorTimeTest_Test::TestBody base/threading/watchdog_unittest.cc:94
    #3  testing::internal::HandleSehExceptionsInMethodIfSupported testing/gtest/src/gtest.cc:2090
  Concurrent read(s) happened at (OR AFTER) these points:
   T3 (L{L670}):
    #0  base::Watchdog::ThreadDelegate::ThreadMain base/threading/watchdog.cc:111 # OR AFTER means that this line number may be a few lines off.
    #1  base::::ThreadFunc base/threading/platform_thread_posix.cc:51
  ...
}}}
Suppression (error hash=#0AE0A877FA5D4A86#):
{
  <Put your suppression name here>  // don't forget to file a bug and replace this line with bug_NNNNN before commiting
  ThreadSanitizer:Race
  fun:base::Watchdog::~Watchdog
  fun:base::::WatchdogCounter::~WatchdogCounter
  fun:base::WatchdogTest_AlarmPriorTimeTest_Test::TestBody
  fun:testing::internal::HandleSehExceptionsInMethodIfSupported
}

Some TSan/threading-related specifics
1) Please don't create large suppressions (i.e. more than 3-4 lines below the top of the stack trace) since they are likely to be interleaving-depended.
Sometimes you don't need ALL the frames to be present in the suppression.
The race reports are very dependent on scheduling, so please take an extra minute to identify the source of the race.
If you think you can see where exactly the synchronization is missing by looking at the racing stacks (believe me you will after investigating 5-10 race reports),
you should replace other frames with "fun:*" (matches one line) or "..." (matches any number of lines).
Otherwise, you'll very likely need to add these wildcards later.

2) A data race report consists of at least two memory accesses:
    * In some case you need to suppress only the first stack trace for each report (the one that starts with T0 in the above example)
    * In some cases you need to prepare suppressions for each stack trace printed in the report (the one that starts with T3 in the above example)

Understanding ThreadSanitizer reports:

General advice: we've tried hard to do the race report format self-descriptive (believe me we've even conducted an UX study for that!)
So please take a minute of your time to read the report carefully - you shouldn't have too many questions after that.

If you still have questions, the ThreadSantizer wiki has a brief description of the race report format.
Feel free to leave a comment there if you still don't understand some particular data race report.

TODO: add more.

Other Valgrind-based race detectors

Valgrind has two other tools that find data races: Helgrind and DRD.
It would be great to get feedback on which of the three race detectors is better.
See the helgrind manual an the DRD manual for more info.
Currently, these two tools are not supported by tools/valgrind/valgrind_test.py but it should be easy to add them if you need.

Comments