Blink‎ > ‎

Importing Tests from the W3C (and contributing back)

The W3C provides a large number of conformance tests for different aspects of the Web Platform. Currently the tests are hosted on GitHub. There are two main repositories, one for the CSS Working Group (aka csswg-test), and one for pretty much everything else (aka web-platform-tests).

Blink has mirrors (web-platform-tests, csswg-test) of the GitHub repos, and periodically imports a subset of the tests so that they are run as part of the regular Blink layout test testing process.

Automatic Importing Process

There is now a w3c-test-autoroller bot which runs the w3c_test_autroller recipe, which attempts to run:

third_party/WebKit/Tools/Scripts/update-w3c-deps --auto-import wpt
third_party/WebKit/Tools/Scripts/update-w3c-deps --auto-import css

Manually Importing New W3C Tests

Updating the set of tests run by Blink requires commit access to Chromium like anything else, so make sure you have that first.

We control which tests are imported via LayoutTests/W3CImportExpectations, which as a blacklist (it is a list of directories and files to skip). This means that any new tests and directories that show up in the W3C repos are automatically imported.

To pull the latest versions of the tests that are currently being imported (i.e., you don't need to change the blacklist), all you have to do is run  Tools/Scripts/update-w3c-deps with css (for csswg-test) or wpt (for web-platform-tests). That script will pull the latest version of the tests from our mirrors of the W3C repos. 

third_party/WebKit/Tools/Scripts/update-w3c-deps wpt
third_party/WebKit/Tools/Scripts/update-w3c-deps css

If any new versions of tests are found, they will be committed locally to your Git repo. You may then upload the changes as a patch to Rietveld like any other patch.

If you wish to add more tests (by un-skipping some of the directories currently in the blacklist), you can modify the file locally, commit it, and then run the update-w3c-deps script with the --allow-local-commits flag. This flag prevents the script from aborting when you have local commits, which is done in order to help ensure the import is done in a reproducible manner, but if you have local commits to W3CImportExpectations, then you need to add this flag.

You can also run the the webkit layout tests including the imported W3C tests at tip-of-tree Chromium to see if there are new failures that will need to be suppressed, or baselines that need to be updated.

In addition to the above, there are some other bugs remaining that limit which tests we can currently import and run, and require us to have to rewrite some of the tests (as noted above) more than we'd like to have to do. We are currently tracking suck bugs in the Chromium issue tracker with the component Blink>Infra.

We also currently do not automatically update the imported tests. We may set up a process for doing so in the future.

Note that currently the main reason we do not run more of the W3C tests is because they are (probably) mostly redundant with Blink's existing tests, and so we would double the running time of the layout tests for little benefit. Ideally we would identify the tests that are redundant and remove Blink's copies, so that we run just the W3C tests where possible.

The end goals for this whole process are to:
  1. Be able to run the W3C tests unmodified locally just as easily as we can run the Blink tests
  2. Ensure that we are tracking tip-of-tree in the W3C repositories as closely as possible
  3. Run as many of the W3C tests as possible
But, we do not have a schedule for accomplishing these goals any time soon. Volunteers to work on this would be most welcome.

Contributing Blink tests back to the W3C

The processes for getting new tests committed the W3C repos are at Some specifics are at

Experimental WPT contribution workflow with Chromium tools

We have another experimental way to contribute web-platform-tests.  It isn't GitHub's fork-modify-pullrequest workflow, but the workflow uses |git cl upload| and Gerrit.  If you don't have any issues on the GitHub workflow, you don't need to use this workflow which this section describes.

* Setup

Install depot_tools, then run the following commands.
% git clone --recursive
(Note: it's NOT
% depot-tools-auth login
TODO(tkent): support "fetch wpt"

* Create a patch

% cd web-platform-tests
% git new-branch branch-name
(edit files ...)
% git commit

* Upload a patch

% git cl upload --gerrit

* Commit a patch

The change should get approval (Code-Review=+2) from one who is a chromium committer and has web-platform-tests push access.
Then, you should change Commit-Queue label to +1.  A bot automatically lands the patch to web-platform-tests repository, and close the code review as 'Abandoned.'  You might be uncomfortable with 'Abandoned' status, however it's by design because we don't merge the change to a git server associated to the Gerrit instance.

* Update your local git repository

% git checkout master
% git pull
% git submodules update --recursive
TODO(tkent): Check if "git rebase-udpate" works, and make it work if not.


  • The bot runs on a personal machine, and isn't stable yet.  Ping tkent@ if your patch isn't committed in one workday.
  • This workflow does't support anything other than web-platform-tests. It doesn't support w3c/csswg-test, w3c/testharness.js, etc.