Layout Tests

Layout tests are used to check the correctness of the renderer.  It involves loading pages in a test renderer (content_shell) and comparing the rendered output tree against an expected output file.  Additionally, there are "pixel tests" which compare an image of the rendered page against an expected file.

Note that Blink is increasingly running more and more of the W3C's conformance tests as part of the Layout Tests. For more information on that, see Running the W3C Tests in Blink .

Getting set up
Before you can run the layout tests, you need to do the following:
  • Build "blink_tests" to get the content_shell and all of the other needed binaries.
    •  $ ninja -C out/Release blink_tests
    • On Android, you need to build and install 'content_shell_apk' instead:
      • $ ninja -C out/Release content_shell_apk
      • $ adb install -r out/Release/apks/ContentShell.apk
  • If on Windows XP, install the necessary fonts (Windows Vista has all the necessary fonts installed by default).  You can do this from Control Panel -> Regional and Language Options -> Languages tab -> check both boxes in the Supplemental languages support section.
  • Install the http://www.apple.com/quicktime plugin or a couple of tests will fail.
  • Make sure that you do not have Apple's Monaco font installed.
  • Make sure the layout tests are checked out in your tree.  They should be located in /src/third_party/WebKit/LayoutTests.  If that directory is empty, check your .gclient file (it's located one directory above your src/ directory), comment out lines that say "src/third_party/WebKit/LayoutTests": None, and re-run gclient sync.
  • If on Windows, run /src/third_party/cygwin/setup_mount.bat to patch the registry so that the cygwin utilities used by the regression know where to find the root of their directory.
  • If on Mac:
    • You probably want to strip the content_shell binary before starting the tests. If you don't, you'll have 5-10 running concurrently, all stuck being examined by the OS crash reporter. This will drag your poor machine to its knees, and probably cause other failures like timeouts where they normally don't occur. 
    • $ strip ./xcodebuild/{Debug,Release}/content_shell.app/Contents/MacOS/content_shell

Running the Tests

The test runner script is in $CHROMIUM_DIR/third_party/WebKit - to run the tests, run:
Tools/Scripts/run-webkit-tests
or on Windows:
python Tools/Scripts/run-webkit-tests
or on Android:
Tools/Scripts/run-webkit-tests --android

Tests marked as SKIP in TestExpectations (in Blink repo) won't be run at all, generally because they cause some intractable tool error. To force one of them to be run, either rename that file or specify the skipped test as the only one on the command line (see below).

Note that currently only the tests listed in SmokeTests are run on the Android bots, since running all layout tests takes too long on Android (and may still have some infrastructure issues).  Most developers focus their blink testing on Linux.  We rely on the fact that the Linux and Android behavior is nearly identical for scenarios outside those covered by the smoke tests.


To run only some of the tests, specify their directories or filenames as arguments to run_webkit_tests.py relative to the layout test directory (/trunk/third_party/WebKit/LayoutTests). For example, to run the fast form tests, use


Tools/Scripts/run-webkit-tests fast/forms


Or you could use


Tools/Scripts/run-webkit-tests fast/fo\*


as a shorthand.


Example: To run the layout tests with a debug build of content_shell, but only test the SVG tests and run pixel tests, you would run Tools/Scripts/run-webkit-tests --debug svg.


As a final quick-but-less-robust alternative, you can also just use the content_shell executable to run specific tests by using (for Windows)


src/chrome/Debug/content_shell.exe --dump-render-tree --no-sandbox full_test_source_path


as in


src/chrome/Debug/content_shell.exe --dump-render-tree --no-sandbox \
    c:/chrome/src/third_party/WebKit/LayoutTests/fast/forms/001.html


but this requires a manual diff against expected results, because the shell doesn't do it for you.


To see a complete list of arguments supported by run-webkit-tests, run run-webkit-tests --help.


Linux Note: We try to match the windows render tree output exactly by matching font metrics and widget metrics.   If there's a difference in the render tree output, we should see if we can avoid rebaselining by improving our font metrics.  For additional information on Linux Layout Tests, please see http://code.google.com/p/chromium/wiki/LayoutTestsLinux.


Mac Note: While the tests are running, a bunch of Appearance settings are overridden for you so the right type of scroll bars, colors, etc. are used.  Your main display's "Color Profile" is also changed to make sure color correction by ColorSync matches what is expected in the pixel tests.  The change is noticeable, how much depends on the normal level of correction for your display.  The tests do their best to restore your setting when done, but if you're left in the wrong state, you can manually reset it by going to System Preferences -> Displays -> Color and selecting the "right" value.

Test Harness Options

This script has a lot of command line flags.  You can pass --help to the script to see a full list of options. A few of the most useful options are below:


 --debug Run the debug build of the test shell (default is release) 
 --build-directory The directory under which to look for the content_shell binary (default is "out" on Linux)
 --verbose Produce more verbose output (eg, list tests that pass) 
 --no-pixel-tests Disable the pixel-to-pixel PNG comparisons and image checksums for tests that don't call layoutTestController.dumpAsText() 
 --reset-resultsWrite all generated results directly into the given directory, overwriting what's there  
 --new-baseline Write all generated results into the most specific platform directory, overwriting what's there. Equivalent to --reset-results --add-platform-expectations
 --startup-dialog Bring up a modal dialog before running the test (useful for attaching a debugger) 
 --child-processes The number of tests to run concurrently (defaults to the number of CPUs; useful with --fully-parallel to split up directories)


For more information on the script, see http://trac.webkit.org/wiki/NewRunWebKitTests

Success and Failure

A test succeeds when its output matches the pre-defined expected results. If any tests fail, the test script will place the actual generated results, along with a diff of the actual and expected results, in src/webkit/{Debug/Release}/layout_test_results/. Look in there for the expected outcome, the actual outcome, as well as a contextual diff between the two. The full expected results are in the src/third_party/WebKit/LayoutTests/platform or alongside their respective tests.

A test that runs but produces the wrong output is marked as "failed", one that causes the test shell to crash is marked "** CRASHED **", and one that takes longer than 10 seconds to complete is aborted and marked "** TIMED OUT **". A row of dots in the script's output indicates one or more tests that passed.


There are some failures that are platform differences between our webkit implementation and Apple's implementation.  That is, we pass the test, but it shows up as a failure because the text diff or the pixel diff are slightly difference.  For example, we render our form controls to look like platform-native widgets while Apple uses Aqua widgets.  In these cases, we want to rebaseline the test by generating new expected results.  See below for instructions on how to rebaseline. 


Known differences with our implementation that we should just rebaseline:

  • Font sizes - We try to match other Windows browser for font sizes.  This sometimes causes text to wrap between different words.  This is also ok.
  • Form controls - On Windows, we use a generic set of widgets rather than aqua styled controls (or the platform's native widgets); on Linux, we use the native widgets. Both of these are different in size and appearance from WebKit's widgets..  Also, becaues of this, our select controls and buttons have a border that show up in the render tree output.  This is also ok.
  • Scrollbars - We use different scrollbar widgets (just like the form controls).  We match the size of Apple's scrollbars so the render tree diff should match.  However, pixel diffs will fail so it's ok to rebaseline the images.
  • V8 exceptions - In general, we try to match javascript exceptions exactly with JSC.  However, sometimes this is hard or undesirable.  If a test is failing only because exception text differs, ask someone on the V8 team if it's something they can change the text of or if we should just rebaseline.
  • GURL difference - GURL tries to be compatible with how IE canonicalizes URLs.  Sometimes, this differs from how KURL canonicalizes.  If you're unsure about whether a test should be rebaselined, ask brettw.

Examples:

Scrollbars/fonts: Apple  vs  Chrome-Win  vs  Chrome-Linux

Form controls: Apple  vs  Chrome-Win  vs  Chrome-Linux

V8 exceptions: Apple  vs  Chrome


Those tests that use NPAPI plugin require the plugin to be installed if you want to view the test from Chrome rather than the content_shell. To install a NPAPI plugin in Chrome, just copy the .dll into plugins directory next to your chrome.exe binary. If the plugins directory does not exist, you can create it.

Test Expectations

The TestExpectations (in Blink repo) file contains the list of all known layout test failures (there are actually multiple files, see the others next to TestExpectations as well). See Test Expectations for more on this.

Virtual Test Suites

The run-webkit-tests harness supports the concept of "virtual test suites". A virtual test suite allows developers to re-run an existing set of tests with different command line arguments. For example, if you were testing a new "blocking repaint" mode in Blink (whatever that would mean), you could make that runtime enabled via a --blocking-repaint flag in content_shell, and then re-run all of the tests in LayoutTests/fast/repaint with the additional flag.

Virtual test suites support having their own baselines, but will fall back to the existing baselines if need be. They also support having their own expectations in TestExpectations.

To add a new virtual test suite, add an entry to the LayoutTests/VirtualTestSuites file. The format for the file is a strict JSON object (no comments) containing a list of suite objects. Each suite object contains three keys, a "prefix" key, a"base" key, and an "args" key. For example, it might contain:

% cat LayoutTests/VirtualTestSuites
[
  ...,
  {
    "prefix": "blocking_repaint",
    "base": "fast/repaint",
    "args": ["--blocking-repaint"],
  },
  ...
]
%

The "prefix" key is a single path component name (i.e., it cannot contain a "/") that will be used for a directory under LayoutTests/virtual . You can think of this as the "name" for a suite, but the prefix does not have to be unique (in case  you want to run multiple directories using the same flags. Using the same prefix for different sets of flags is not recommended. 

The "base" key indicates the subpath under LayoutTests/ to the directory of tests you want to run (e.g., "fast/repaint" in the above example). 

The "args" key is a list of the additional command line arguments to use. 

Thus, interpreting the above example, if you have a test in fast/repaint called foo.html, running on the Mac the test harness would act as if it also found a LayoutTests/virtual/blocking_repaint/fast/repaint/foo.html test. It would look for baselines in LayoutTests/platform/mac/virtual/fast/repaint, then LayoutTests/virtual/fast/repaint, then LayoutTests/platform/mac/fast/repaint, then LayoutTests/fast/repaint.

Note that virtual test suites do not inherit any test expectations from the underlying file; if TestExpectations contains "Bug(x) fast/repaint/foo.html [ Fail ]", the virtual file will still be expected to pass. If you want the virtual test to *also* be expected to fail, you have to add another suppression.

Tracking Test Failures

All bugs, associated with layout test failures must have the LayoutTests label. Depending on how much you know about the bug, assign the status accordingly:

  • Unconfirmed -- you aren't sure if this is a simple rebaseline, possible duplicate of an existing bug, or a real failure
  • Available -- you know the root cause of the issue.
  • Assigned or Started -- you will fix this issue.

When creating a new layout test bug, please assign these labels to it -- having proper label hygiene is good for everyone:

  • Type-Bug
  • Pri-2 (Pri-1 if it's a crash)
  • Area-WebKit
  • OS-All (or whichever OS the failure is on)
  • LayoutTests
  • Mstone-9 (or current milestone)
  • Tests-Flaky (if the test is flaky)
You can also use Layout Test Failure template, which will pre-set these labels for you.

Tests that use a HTTP Server

HTTP tests use a locally running HTTP server to run (lighttpd on Windows, Apache elsewhere). You don't need any special configuration to run these, but you should make sure the server is not already running. As a rule, if the test is in an http directory, it will run off the server. If you want to run various tests manually while the local http server is running, note that it exposes different test suites on different ports. For example, all tests are available via port 8081, but the webkit tests are available via port 8000 (8443 for SSL) and the pending tests are available via port 9000 (9443 for SSL).


If you run the tests using run_webkit_tests.py, the server will be started automatically. To run the server manually:


cd src/third_party/WebKit/Tools/Scripts

run-blink-httpd start


The layout tests will be served from http://localhost:8000.


For convenience, you can also use src/webkit/tools/layout_tests/run_http_server.sh to start and stop the server which calls through to the run-blink-httpd script in the Tools/Scripts directory.


To kill the server, run run-blink-httpd --server stop, or just use taskkill or the Task Manager on Windows, and killall or Activity Monitor on MacOS.


(Note that you need to have python in your path to run this by hand; we don't provide a wrapper script.)

If you get errors while running the tests that look like this:

  File "~/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/http_server.py", line 213, in start

    self._process = subprocess.Popen(start_cmd, env=env, stdin=subprocess.PIPE)

  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__

    errread, errwrite)

  File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child

    raise child_exception

OSError: [Errno 2] No such file or directory


...then you probably don't have lighttpd installed.  On linux, try "sudo apt-get install lighttpd".

The test server sets up an alias to LayoutTests/resources directory. In HTTP tests, you can access testing framework at e.g. "/js-test-resources/js-test.js". Use it instead of putting lots of ".."s.

Debugging a test

After the layout tests run, you should get a summary of tests that pass or fail.  If something fails unexpectedly (a new regression), you will get a content_shell window with a summary of the unexpected failures. Or you might have a failing test in mind to investigate. In any case, here are some steps and tips for finding the problem.
  1. Take a look at the result. Sometimes tests just need to be rebaselined (see below) to account for changes introduced in your patch.
    • Load the test into a trunk Chrome or content_shell build and look at its result. (For tests in the http/ directory, start the http server first. See above. Navigate to http://localhost:8000/ and proceed from there.) The best tests describe what they're looking for, but not all do, and sometimes things they're not explicitly testing are still broken.  Compare it to Safari, Firefox, and IE if necessary to see if it's correct. If you're still not sure, find the person who knows the most about it and ask.
    • Some tests only work properly in content_shell, not Chrome, because they rely on extra APIs exposed there.
    • Some tests only work properly when they're run in the layout-test framework, not when they're loaded into content_shell directly. The test should mention that in its visible text, but not all do. So try that too. See "Running the tests", above.
  2. If you think the test is correct, confirm your suspicion by looking at the diffs between the expected result and the actual one.
    • Make sure that the diffs reported aren't important. Small differences in spacing or box sizes are often unimportant, especially around fonts and form controls. Differences in wording of JS error messages are also usually acceptable.
    • $ ./run_webkit_tests.py path/to/your/test.html --full-results-html will produce a page including links to the expected result, actual result, and diff.
    • Add the --sources option to run_webkit_tests.py to see exactly which expected result it's comparing to (a file next to the test, something in platform/mac/, something in platform/chromium-win/, etc.)
    • If you're still sure it's correct, rebaseline the test (see below).  Otherwise...
  3. If you're lucky, your test is one that runs properly when you navigate to it in content_shell normally. In that case, build the Debug content_shell project, fire it up in your favorite debugger, and load the test file either from a file:// URL.
    • You'll probably be starting and stopping the content_shell a lot. In VS, to save navigating to the test every time, you can set the URL to your test (file: or http:) as the command argument in the Debugging section of the content_shell project Properties.
    • If your test contains a JS call, DOM manipulation, or other distinctive piece of code that you think is failing, search for that in the Chrome solution. That's a good place to put a starting breakpoint to start tracking down the issue.
    • Otherwise, you're running in a standard message loop just like in Chrome.  If you have no other information, set a breakpoint on page load.
  4. If your test only works in full layout-test mode, or if you find it simpler to debug without all the overhead of an interactive session, start the content_shell with the command-line flag --dump-render-tree, followed by the URL (file: or http:) to your test. More information about running layout tests in content_shell can be found here.
    • In VS, you can do this in the Debugging section of the content_shell project Properties.
    • Now you're running with exactly the same API, theme, and other setup that the layout tests use.
    • Again, if your test contains a JS call, DOM manipulation, or other distinctive piece of code that you think is failing, search for that in the Chrome solution. That's a good place to put a starting breakpoint to start tracking down the issue.
    • If you can't find any better place to set a breakpoint, start at the TestShell::RunFileTest() call in content_shell_main.cc, or at shell->LoadURL() within RunFileTest() in content_shell_win.cc.
  5. Debug as usual. Once you've gotten this far, the failing layout test is just a (hopefully) reduced test case that exposes a problem.

Tips

  • Check http://test-results.appspot.com/ to see how a test did in the most recent ~100 builds on each builder (as long as the page is being updated regularly).
  • A timeout will often also be a text mismatch, since the wrapper script kills the content_shell before it has a chance to finish. The exception is if the test finishes loading properly, but somehow hangs before it outputs the bit of text that tells the wrapper it's done.
  • Why might a test fail (or crash, or timeout) on the buildbot, but pass on your local machine?
    • If the test finishes locally but is slow, more than 10 seconds or so, that would be why it's called a timeout on the bot.
    • Otherwise, try running it as part of a set of tests; it's possible that a test one or two (or ten) before this one is corrupting something that makes this one fail.
    • If it consistently works locally, make sure your environment looks like the one on the bot (look at the top of the stdio for the webkit_tests step to see all the environment variables and so on).
    • If none of that helps, and you have access to the bot itself, you may have to log in there and see if you can reproduce the problem manually.

Rebaselining Tests


To automatically re-baseline tests across all Chromium platforms, using the buildbot results, see the Rebaselining keywords in TestExpectations and  Rebaselining Tool. Alternatively, to manually run and test and rebaseline it on your workstation, read on.


By default, text-only tests (ones that call layoutTestController.dumpAsText()) produce only text results. Other tests produce both new text results and new image results (the image baseline comprises two files, -expected.png and -expected.checksum). So you'll need either one or three -expected.* files in your new baseline, depending on whether you have a text-only test or not. If you enable the --no-pixel-tests, only new text results will be produced, even for tests that do image comparisons.


cd src/third_party/WebKit

Tools/Scripts/run-webkit-tests --new-baseline foo/bar/test.html


The above command will generate a new baseline for LayoutTests/foo/bar/test.html and put the output files in the right place, e.g.  src/third_party/WebKit/LayoutTests/platform/chromium-win/LayoutTests/foo/bar/test-expected.{txt,png,checksum}


When you rebaseline a test, make sure your commit description explains why the test is being re-baselined.  If this is a special case (i.e., something we've decided to be different with upstream), please put a README file next to the new expected output explaining the difference.

Known Issues

  • Windows & Linux : Do not copy and paste while the layout tests are running, as it may interfere with the editing/pasteboard and other clipboard-related tests (Mac tests swizzle NSClipboard to avoid any conflicts).
  • If QuickTime is not installed, the plugin tests fast/dom/object-embed-plugin-scripting.html and plugins/embed-attributes-setting.html are expected to fail.

TestRunner LayoutTest support

Some layout tests rely on the testRunner object to expose configuration for mocking the platform. This is provided in content_shell, here's a UML of testRunner bindings configuring platform implementation.