There is a Google-grown Valgrind tool called ThreadSanitizer (aka TSan), which finds data races. We have three ThreadSanitizer build bot slaves set up on the main waterfall for Windows, Linux and Mac. Also, we have a ThreadSanitizer UI bot on the FYI waterfall. If curious, see the list of Chromium races found by ThreadSanitizer. One of the most scary races found by ThreadSanitizer is a data race on reference counter which caused dozens of other bugs. Together, these bugs were the reason of chromium top crasher for several months! It took less than an hour to detect the race with ThreadSanitizer while it was completely overlooked for months of testing and debugging. Running ThreadSanitizer on Chromium (Linux and Mac)In order to use it, you'll need to get ThreadSanitizer binaries from SVN.Then, make sure you have your local GYP flags overriden like you should do for Valgrind (see using-valgrind). To run Chromium tests under ThreadSanitizer, simply add "--tool tsan" to your Valgrind command, e.g.: sh tools/valgrind/chrome_tests.sh -t base --tool tsan See the Thread Sanitizer home page for more info on the tool itself. Please note that by default ThreadSanitizer runs on Chromium in conservative mode (--pure-happens-before=yes). You can pass additional flags to ThreadSanitizer by using --tool_flags="--some-tsan-flag" Running ThreadSanitizer on Chromium (Windows)Most of the Linux and Mac instructions are still valid but there are specifics.To get the binaries from SVN please add the following line to your .gclient file instead of using Valgrind instructions: { "name" : "src", "url" : <...>, "custom_deps" : { <your old custom_deps here> "src/third_party/tsan": "http://src.chromium.org/chrome/trunk/deps/third_party/tsan", },You can only run Debug builds under TSan on Windows. They can be slow, so to speed them up the following GYP values are recommended: { 'variables': { 'build_for_tool': 'tsan', }, } or just set GYP_DEFINES="build_for_tool=tsan" gclient runhooks Now you can run your Chromium test using (notice .bat instead of .sh): tools\valgrind\chrome_tests.bat -t base --tool tsan Suppressing data racesThreadSanitizer suppression format is very similar to that of Memcheck, so you can create suppressions the same way as you do for Memcheck.Good news: ThreadSanitizer also supports demangled suppressions! The ThreadSanitizer suppression file are located at tools/valgrind/tsan/suppression.txt and tools/valgrind/tsan/suppression_{linux,mac}.txt The ThreadSanitizer runner script generates suppressions: WARNING: Possible data race during write of size 8 at 0x7FFEFFF7D8: {{{ T0 (L{}): #0 base::Watchdog::~Watchdog base/threading/watchdog.cc:33 #1 base::::WatchdogCounter::~WatchdogCounter base/threading/watchdog_unittest.cc:28 #2 base::WatchdogTest_AlarmPriorTimeTest_Test::TestBody base/threading/watchdog_unittest.cc:94 #3 testing::internal::HandleSehExceptionsInMethodIfSupported testing/gtest/src/gtest.cc:2090 Concurrent read(s) happened at (OR AFTER) these points: T3 (L{L670}): #0 base::Watchdog::ThreadDelegate::ThreadMain base/threading/watchdog.cc:111 # OR AFTER means that this line number may be a few lines off. #1 base::::ThreadFunc base/threading/platform_thread_posix.cc:51 ... }}} Suppression (error hash=#0AE0A877FA5D4A86#): { <Put your suppression name here> // don't forget to file a bug and replace this line with bug_NNNNN before commiting ThreadSanitizer:Race fun:base::Watchdog::~Watchdog fun:base::::WatchdogCounter::~WatchdogCounter fun:base::WatchdogTest_AlarmPriorTimeTest_Test::TestBody fun:testing::internal::HandleSehExceptionsInMethodIfSupported } Some TSan/threading-related specifics 1) Please don't create large suppressions (i.e. more than 3-4 lines below the top of the stack trace) since they are likely to be interleaving-depended. Sometimes you don't need ALL the frames to be present in the suppression. The race reports are very dependent on scheduling, so please take an extra minute to identify the source of the race. If you think you can see where exactly the synchronization is missing by looking at the racing stacks (believe me you will after investigating 5-10 race reports), you should replace other frames with "fun:*" (matches one line) or "..." (matches any number of lines). Otherwise, you'll very likely need to add these wildcards later. * In some cases you need to prepare suppressions for each stack trace printed in the report (the one that starts with T3 in the above example) Understanding ThreadSanitizer reports:General advice: we've tried hard to do the race report format self-descriptive (believe me we've even conducted an UX study for that!) If you still have questions, the ThreadSantizer wiki has a brief description of the race report format. TODO: add more. Other Valgrind-based race detectorsValgrind has two other tools that find data races: Helgrind and DRD.It would be great to get feedback on which of the three race detectors is better. See the helgrind manual an the DRD manual for more info. Currently, these two tools are not supported by tools/valgrind/valgrind_test.py but it should be easy to add them if you need. |
ThreadSanitizer (TSAN) |