Blink‎ > ‎

Importing Tests from the W3C (and contributing back)

The W3C provides a large number of conformance tests for different aspects of the Web Platform. Currently the tests are hosted on GitHub, two main repositories, one for the CSS Working Group (aka csswg-test), and one for pretty much everything else (aka web-platform-tests).

Blink has mirrors (web-platform-tests, csswg-test) of the GitHub repos, and periodically imports a subset of the tests so that they are run as part of the regular LayoutTests testing process (the tests are pulled in via an entry to DEPS to LayoutTests/w3c).

This page documents the current process for modifying the subset that is imported and run, and how we can submit new tests back to the W3C.

For some historical discussion of how things work (written just after the Blink fork from WebKit), see Blink, Testing, and the W3C .

Importing new W3C Tests into Blink

Updating the set of tests run by Blink requires commit access to Blink like anything else, so make sure you have that first.

We control which tests are imported via LayoutTests/W3CImportExpectations, which as a blacklist (it is a list of directories and files to skip). This means that any new tests and directories that show up in the W3C repos are automatically imported.

To pull the latest versions of the tests that are currently being imported (i.e., you don't need to change the blacklist), all you have to do is run  Tools/Scripts/update-w3c-deps with css (for csswg-test) or wpt (for web-platform-tests). That script will pull the latest version of the tests from our mirrors of the W3C repos. 

% ./third_party/WebKit/Tools/Scripts/update-w3c-deps css
% ./third_party/WebKit/Tools/Scripts/update-w3c-deps wpt

If any new versions of tests are found, they will be committed locally to your Git repo. You may then upload the changes as a patch to Rietveld like any other patch.

If you wish to add more tests (by un-skipping some of the directories currently in the blacklist), you can modify the file locally, commit it, and then run the update-w3c-deps script with the --allow-local-blink-commits flag (normally we don't want to allow local commits, to help ensure that the import is done in a reproducible manner).

You can also run the tests locally against tip-of-tree Blink and/or Chromium's currently pinned version to see if there are new failures that will need to be suppressed, or baselines that need to be updated.

In addition to the above, there are some other bugs remaining that limit which tests we can currently import and run, and require us to have to rewrite some of the tests (as noted above) more than we'd like to have to do. We are currently tracking such bugs in the Chromium issue tracker using the label Cr-Blink-Tools-Test-W3C.

We also currently do not automatically update the imported tests. We may set up a process for doing so in the future.

Note that currently the main reason we do not run more of the W3C tests is because they are (probably) mostly redundant with Blink's existing tests, and so we would double the running time of the layout tests for little benefit. Ideally we would identify the tests that are redundant and remove Blink's copies, so that we run just the W3C tests where possible.

The end goals for this whole process are to:
  1. Be able to run the W3C tests unmodified locally just as easily as we can run the Blink tests
  2. Ensure that we are tracking tip-of-tree in the W3C repositories as closely as possible
  3. Run as many of the W3C tests as possible
But, we do not have a schedule for accomplishing these goals any time soon. Volunteers to work on this would be most welcome.

Contributing Blink tests back to the W3C

As described above, the W3C has two main repositories of tests, one for the CSS working group, and one for everything else. The processes for getting new tests committed to each repo are documented here for web-platform-tests and here for csswg-test.

This section describes how to take tests currently checked into LayoutTests/ and get them added to the W3C repos. If you want to submit tests that are not part of LayoutTests/, you're on your own (but can probably follow the existing W3C documentation).

We currently have Blink specific forks of the W3C repos hosted on GitHub (web-platform-tests and csswg-test), in order to save individual Blink contributors from having to create their own individual forks. For access to the Blink repos, talk to the infrastructure team.

Once you have access, you should push a change to the Blink repo, and then create a Pull request to the W3C repo. Then get one of the W3C committers (TBD: who??) to grant the pull request and land the change. 

If the new tests are currently being run in Blink (i.e., they are checked into LayoutTests somewhere), then in theory that should count as "reviewed" and getting the pull request approved should be straightforward. 

Once the PR has landed in the W3C's repo, you can then refer to the "Importing new W3C Tests into Blink" section above to get the tests pulled back into Blink, at which point you can then finally delete the copies checked in under LayoutTests/.