Test Expectations

The primary function of the LayoutTests is as a "regression test suite". This means that, while we care about whether a page is being rendered correctly, we care more about whether the page is being rendered the way we expect it to. In other words, we look more for changes in behavior than we do for correctness.

All layout tests have "expected results", which may be one of several forms. The test may produce one or more of

  • a text file containing javascript log messages
  • a text rendering of the Render Tree. 
  • a screen capture of the rendered page as PNG files as well. 
  • a WAV files of the audio output for WebAudio tests instead of either text or PNG files. 
For any of these types of tests, there are files checked into the LayoutTests directory named "-expected.{txt,png,wav}". 

Lastly, we also support the concept of "reference tests", which check that two pages are rendered identically (pixel-by-pixel). As long as the two tests' output match, the tests pass. For more on reference tests, see http://trac.webkit.org/wiki/Writing%20Reftests.

When the output doesn't match, there are two potential reasons for it:

  1. The port is performing "correctly", but the output simply won't match the generic version. The usual reason for this is for things like form controls, which are rendered differently on each platform.
  2. The port is performing "incorrectly" (i.e., the test is failing).
In both cases, the convention is to check in a new "-expected" file, even though that file may be codifying errors. This helps us maintain test coverage for all the other things the test is testing while we resolve the bug.

What do to with a failing test

If a test can be rebaselined, it should always be rebaselined instead of adding lines to TestExpectations. Bugs at crbug.com should track fixing incorrect behavior, not lines in TestExpectations.

If a test is never supposed to pass (e.g. it's testing Windows-specific behavior, so can't ever pass on Linux/Mac), move it to the NeverFixTests file. That gets it out of the way of the rest of the project.

There are some cases where you can't rebaseline and, unfortunately, we don't have a better solution than reverting the patch that caused the failure or adding a line to TestExpectations and fixing the bug later. Reverting the patch is strongly preferred.

These are the cases where you can't rebaseline:
  • If the test is a reftest
  • If the test gives different output in release and debug; in this case, generate a baseline with the release build, and mark the debug build as expected to fail
  • If the test is flaky, crashes or times out.
  • If the test is for a feature that isn't shipped on some platforms yet, but will shortly.

Handling flaky tests

The flakiness dashboard is a tool for understanding a test’s behavior over time. Originally designed for managing flaky tests, the dashboard shows a timeline view of the test’s behavior over time. The tool may be overwhelming at first, but the documentation should help. Once you decide that a test is truly flaky, you can suppress it using the TestExpectations file, below.

We do not generally expect chromium sheriffs to spend time trying to address flakiness, though.

How to rebaseline

  • If the change has not yet landed or cycled on the build bots:
    • If the test is already listed in TestExpectations as flaky:
      • Mark the test NeedsManualRebaseline and comment out the flaky line so that you patch can land without turning the tree red.
      • Once the bots have cycled, follow the rebaseline-expectations steps below and uncomment out the flaky line commented out in the previous checkin.
    • If the test is not already listed in TestExpectations, mark it as NeedsRebaselineThe rebaseline-o-matic bot will automatically detect when the bots have cycled (by looking at the blame on the file) and do the rebaseline for you. As long as the test doesn't timeout or crash, it won't turn the bots red if it has a NeedsRebaseline expectation.
  • If the change has already cycled on all the build bots:
    1. Add a [ Rebaseline ] line to TestExpectations
    2. run "webkit-patch rebaseline-expectations"
    3. Post the patch created in step 2 for review

A bot called the Auto-Rebaseline-Bot (or rebaseline-o-maticperiodically runs, looking for TestExpectations entries that [ NeedsRebaseline ]. The bot gathers new baselines when they are available, and automatically lands a CL with the new baselines for you to review. So If you see TestExpectations entries with [ NeedsRebaseline ], ignore them. If you don't see a bot periodically removing these lines from TestExpectations and landing baselines, bug someone via blink-dev@.

If you see [ NeedsManualRebaseline ] entries, this is basically equivalent to "Failure" but indicates that someone (the developer who made the change) needs to manually review and update the baselines. You should also ignore these entries, or nag the people who added them to clean up after themselves if you're bored.

Kinds of expectations files

  • ASANExpectations: Tests that fail under ASAN
  • LeakExpectations: Tests that have memory leaks under the leak checker
  • MSANExpectations: Tests that fail under MSAN
  • NeverFixTests: Tests that we never intend to fix (e.g. a test for Windows-specific behavior will never be fixed on Linux/Mac). Tests that will never pass on any platform should just be deleted, though.
  • OilpanExpectations: Tests that only fail in the non-oilpan configuration (yes, this naming is confusing. This file will go away soon).
  • OilpanLeakExpectations: Tests that only fail in the non-oilpan configuration under the leak checker (This too).
  • SlowTests: Tests that take longer than the usual timeout to run. Slow tests are given 5x the usual timeout.
  • SmokeTests: A small subset of tests that we run on the Android bot.
  • StaleTestExpectations: Platform-specific lines that have been in TestExpectations for many months. They're moved here to get them out of the way of people doing rebaselines since they're clearly not getting fixed anytime soon.
  • TestExpectations: The main test failure suppression file. This should really only be used for flaky lines and NeedsRebaseline/NeedsManualRebaseline lines. 
  • W3CImportExpectations: Expectations for auto-imported w3c tests.
  • WPTServeExpectations: Expectations for tests that fail differently when run under the W3C's wptserve HTTP server, rather than Apache (this is an experimental feature at this time).

Flag-specific expectations files

It is possible to handle tests that only fail when run with a particular flag being passed to content_shell. See https://chromium.googlesource.com/chromium/src/+/master/third_party/WebKit/LayoutTests/FlagExpectations/README.txt for more.

Updating the expectations files


The file is not ordered. Hint: Put new changes into a random spot in the file to reduce the chance of merge conflicts when landing your patch.


The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with "# " are treated as comments, and blank lines are allowed as well.

The syntax of a line is roughly:

[ bugs ] [ "[" modifiers "]" ] test_name [ "[" expectations "]" ]
  • Tokens are separated by whitespace.
  • The brackets delimiting the modifiers and expectations from the bugs and the test_name are not optional; however modifiers components is optional. In other words, if you want to specify modifiers or expectations, you must enclose them in brackets.
  • Lines are expected to have one or more bug identifiers, and the linter will complain about lines missing them. Bug identifiers are of the form "crbug.com/12345", "code.google.com/p/v8/issues/detail?id=12345" or "Bug(username)".
  • If no modifiers are specified, the test applies to all of the configurations applicable to that file.
  • Modifiers can be one or more of Mac, Mac10.6, Mac10.7, Mac10.8, Mac10.9Mac10.10, RetinaWin, Win7Win10Linux, Linux32, Precise, Trusty, AndroidRelease, Debug.
  • Some modifiers are meta keywords, e.g. Win represents both Win7 and Win10. See the CONFIGURATION_SPECIFIER_MACROS dictionary in Tools/Scripts/webkitpy/layout_tests/port/base.py for the meta keywords and which modifiers they represent.
  • Expectations can be one or more of Crash, Failure, Pass, Rebaseline, Slow, Skip, Timeout, WontFix, Missing,NeedsRebaseline,NeedsManualRebaseline. If multiple expectations are listed, the test is considered "flaky" and any of those results will be considered as expected.
For example:

crbug.com/12345 [ Win Debug ] fast/html/keygen.html [ Crash ]

which indicates that the "fast/html/keygen.html" test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the chromium issue tracker. Note that the test will still be run, so that we can notice if it doesn't actually crash.

Assuming you're running a debug build on Mac Lion, the following lines are all equivalent (in terms of whether the test is performed and its expected outcome):

fast/html/keygen.html [ Skip ]
fast/html/keygen.html [ WontFix ]
Bug(darin) [ Lion Debug] fast/html/keygen.html [ Skip ]


  • WontFix implies Skip and also indicates that we don't have any plans to make the test pass.
  • WontFix lines go in the NeverFixTests file as we never intend to fix them. These are just for tests that only apply to some subset of the platforms we support.
  • WontFix and Skip must be used by themselves and cannot be specified alongside Crash or another expectation keyword.
  • Slow causes the test runner to give the test 5x the usual time limit to run. Slow lines go in the SlowTests file. A given line cannot have both Slow and Timeout.
Also, when parsing the file, we use two rules to figure out if an expectation line applies to the current run:
  1. If the configuration parameters don't match the configuration of the current run, the expectation is ignored.
  2. Expectations that match more of a test name are used before expectations that match less of a test name.
For example, if you had the following lines in your file, and you were running a debug build on Mac SnowLeopard:

crbug.com/12345 [ SnowLeopard ] fast/html [ Failure ]
crbug.com/12345 [ SnowLeopard ] fast/html/keygen.html [ Pass ]
crbug.com/12345 [ Win7 ] fast/forms/submit.html [ Failure ]
crbug.com/12345 fast/html/section-element.html [ Failure Crash ]

You'd expect:
  • fast/html/article-element.html to fail with a text diff (since it is in the fast/html directory).
  • fast/html/keygen.html to pass (since the exact match on the test name).
  • fast/html/submit.html to pass (since the configuration parameters don't match).
  • fast/html/section-element.html to either crash or produce a text (or image and text) failure, but not time out or pass.
Again, duplicate expectations are not allowed within the file and will generate warnings. 

You can verify that any changes you've made to an expectations file are correct by running:

% lint-test-expectations

which will cycle through all of the possible combinations of configurations looking for problems.