For Developers‎ > ‎How-Tos‎ > ‎

Using Dr. Memory

Dr. Memory is a new open-source tool for finding memory errors - similar to Memcheck (Valgrind) but with Windows support, including detection of Windows handle leaks and GDI usage errors.

Dr. Memory is running on the MFYI waterfall MFYI waterfall where it finds bugs in Chromium on Windows.

If you want to try Dr. Memory on your machine, please do the following:
  • First of all, you need Windows.  Win7 x64 is preferred.  DrMemory supports Linux, but the support is not as robust.
  • Build Chromium tests on Windows in either Release or Debug configuration (you can save some time by building only the small tests like googleurl, ipc, base, net to start with).  Release runs faster.  For either Release or Debug, build with the following setting in order to disable inlining and FPO for better callstacks by placing this in your ~/.gyp/include.gypi file:
    {
      'variables': {
        'build_for_tool': 'drmemory',
      },
    }
    or
      set GYP_DEFINES=build_for_tool=drmemory
    gclient runhooks

  • The Dr. Memory binaries are downloaded from Google Storage automatically (on Windows) into the third_party/drmemory directory.

  • Now you can run chromium tests like this:
    tools\valgrind\chrome_tests.bat -t <test> --tool drmemory_light # only search for unaddressable accesses and uses-after-free
    or
    tools\valgrind\chrome_tests.bat -t <test> --tool drmemory_full  # search for uninits and leaks etc as well
    Alternatively, if you prefer Cygwin, you can also use tools/valgrind/chrome_tests.sh with the same arguments.  Just be aware that the bots use the .bat script.

  • You can use --build-dir to select between Debug and Release, for example: tools\valgrind\chrome_tests.bat -t base_unittests --tool drmemory_light --build-dir=out\Release
  • It will print you some error reports at the end.
If for some reason you cannot reproduce a bot failure on your Windows machine, look for the exact command that started the tests and run that. For example, here is one variation of the test command that catches some memory bugs that the above commands may not catch:

  src\tools\valgrind\chrome_tests.bat --brave-new-test-launcher --test-launcher-bot-mode --test <test> --tool drmemory_full --target Release --build-dir src\out

OK, I've found a real bug

Great!
Please file a crbug and add the Stability-DrMemory label.

drmemory_light vs drmemory_full

Light: searches for unaddressable accesses like OOB or use-after-frees, handle leaks, and graphical library usage errors, incurring a moderate execution slowdown.
Full: additionally searches for uninitialized reads but adds a large slowdown.

More on the "full" mode:
If you're familiar with the Valgrind guts you probably know that tools like Valgrind and Dr. Memory need to intercept and handle all system calls to know precisely what data is uninitialized, what is read and so on.
In contrast to the open-source Linux kernel with just a hundred system calls or so, on Windows we have an open-ended problem of handling undocumented system calls.
We're trying hard to do that but we haven't finished yet.  We have some heuristics that work pretty well to ignore errors stemming from system libraries making system calls as opposed to Chromium code, but if you do see an error report that you believe is a false positive, please create a short reproducer and file it into our issue tracker.

Suppressing error reports from the bots

TODO(rnk): When the DrMemory bots have moved to the chromium.memory.fyi waterfall, this information will be moved into the memory sheriff page.
timurrrr: most of this info sounds DrMemory-specific, do we really want to put it onto the "common" page?

The DrMemory suppression file is at tools/valgrind/drmemory/suppressions[_full].txt.
Generally speaking, suppressions in these files are maintained in exactly the same way as those for all of the other memory tools we use for Chromium, except that DrMemory uses a slightly different suppression format.

When the bot generates a report, follow the link to the bot's stdio and search for "hash=#" to find the report quickly.  You should see something similar to:
UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s)
# 0 TestingProfile::FinishInit           [chrome\test\base\testing_profile.cc:211]
# 1 TestingProfile::TestingProfile       [chrome\test\base\testing_profile.cc:164]
# 2 BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody  [chrome\browser\browser_about_handler_unittest.cc:110]
# 3 testing::Test::Run                   [testing\gtest\src\gtest.cc:2162]
Note: @0:00:08.656 in thread 1900
Note: instruction: mov    (%ebx) -> %esi
Suppression (error hash=#30459DB8320B1FA3#):
{
UNADDRESSABLE ACCESS
name=<insert_a_suppression_name_here>
*!TestingProfile::FinishInit
*!TestingProfile::TestingProfile
*!BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody
*!testing::Test::Run
}

As a starting point, you can copy the suppression (the portion between the curly brackets not including the brackets) into the suppressions.txt file and then widen it so it covers any similar reports on other bots.
The curly brackets are not part of the suppression as in all the other memory tools! DrMemory suppressions are separated by blank lines instead, so make sure your suppression does not run into an existing suppression.

Each frame from the stack trace is of the form "<module>!<function>". For functions from Chrome, we generally link the code statically into many different binaries, so we always use a wildcard for the module.
For third party code, it is generally helpful to identify which DLL the frame came from, especially since we often do not have debug info for windows system libraries.

As in the other tool suppression formats, '*' is a single frame variable-width wildcard, '?' is a single-character wildcard, and '...' will match any number of frames.

DrMemory has support for a handful of other pattern matching mechanisms.
In particular, if you paste the instruction from the report on a line starting with "
instruction=", you can match the exact instruction producing the report.
The
instruction= qualifier is mostly useful when suppressing bit-level false positive uninitialized reads in third party libraries where we don't have an accurate callstack and want to make a tight suppression.

Once you have your suppression, you can test that it suppresses the existing reports using tools/valgrind/waterfall.sh as you would with the other memory tools.

Running Chromium under Dr. Memory on a given set of URLs

First, put the list of URLs into tools\valgrind\reliability\url_list.txt, one URL per line without the http:// prefix. You can also put in the local URLs like file://c:\mypath\repro.htm
Then, run
tools\valgrind\chrome_tests.bat -t reliability --tool drmemory --tool_flags="-no_check_uninitialized -no_count_leaks"

If you're searching for some stability/security bugs, you may consider running Chromium in the single process mode to increase the probability of the report by applying this patch:
Index: tools/valgrind/chrome_tests.py
===================================================================
--- tools/valgrind/chrome_tests.py (revision 96002)
+++ tools/valgrind/chrome_tests.py (working copy)
@@ -271,7 +271,8 @@
# Valgrind timeouts are in seconds.
UI_VALGRIND_ARGS = ["--timeout=7200", "--trace_children", "--indirect"]
# UI test timeouts are in milliseconds.
- UI_TEST_ARGS = ["--ui-test-action-timeout=120000",
+ UI_TEST_ARGS = ["--single-process=yes",
+ "--ui-test-action-timeout=120000",
"--ui-test-action-max-timeout=280000"]

def TestAutomatedUI(self):

Running your custom command line under Dr. Memory

Sometimes you may need to run your own program or your own custom command line under Dr. Memory. For example, if you want to prefix your own "chrome.exe --my-flags" command with Dr. Memory, do the following:
tools\valgrind\drmemory.bat myapp.exe --my-flags  # don't forget to increase the timeouts
If the command line runs Chrome, you may consider increasing the timeouts by appending
--ui-test-action-timeout=120000 --ui-test-action-max-timeout=280000 --ui-test-terminate-timeout=120000
to the commandline.

Random notes for the curious ones
You can find the Dr. Memory suppression file for Chrome at tools\valgrind\drmemory\suppressions[_full].txt

The report files are parsed and re-formatted to be more familiar for Valgrind users (see tools\valgrind\drmemory_analyze.py).

Excluded tests are listed in tools\valgrind\gtest_exclude - see Using Valgrind more for information.

Feedback?

Drop drmemory-team@ a message

Comments