Android Tests

Concepts

  • Native Unit Tests - Normal Chromium unit tests based on the gtest framework.
  • Instrumentation Tests - Java tests written using the Android Instrumentation Test Framework.
  • Unit Instrumentation Tests - Instrumentation tests that only tests Java code and not native code. Do not require JNI bindings and do not require starting up ContentShell (or another application). These should extend InstrumentationTestCase directly.
  • Integration Instrumentation Tests - Instrumentation tests that tests the entire stack from Java into native code, require JNI bindings to be set up and thus require ContentShell (or another application) to be started. These typically extend more specialized test base classes such as ContentShellTestBase.

Running Tests

View the guide on the Google Code page for running Android tests: https://code.google.com/p/chromium/wiki/AndroidTestInstructions

Location of Test Code

As a general rule, tests should live next to the production code and test utilities should live in a test folder.

In general, this should look like:
  • foo/android/java/src/bar - production code.
  • foo/android/javatests/src/bar - unit instrumentation tests.
  • foo/test/android/javatests/src/bar/test/util - reusable test utilities used to test the foo production code.
  • native unit tests should live next to the source (same as general Chromium unit test)
Test utilities that are pure Java and do not depend on content or other Chromium code should live in:
  • base/test/android/javatests/src/org/chromium/base/test/util
Test utilities in the content layer that are re-used by chrome should live in:
  • content/public/test/android/javatests/src/org/chromium/content/test/util
The location of integration instrumentation tests depends on what they are testing:
  • If the test is directly testing code in content/public/android/java/src, it should live in content/public/android/javatests/src (although it might require the test base class from content/shell/android/javatests/src).
  • If the test is directly testing code in chrome/android/java/src, it should live in chrome/android/javatests/src (although it might require the test base class from chrome/android/testshell/javatests/src).
  • If the test is a general integration test (i.e. testing general functionality that might not be tied directly to specific code), it should live in the appropriate javatests folder:
    • content/shell/android/javatests/src - Content Shell
    • chrome/android/testshell/javatests/src - Chrome Test Shell
    • android_webview/javatests/src - Android WebView
Please note that this implies that outside of Content Shell, Chrome Test Shell and Android WebView, you should:
  • write unit instrumentation tests to test your Java code.
  • write native tests for everything else.

Guidelines for Writing Tests

TestBase Classes and Test Utilities

  • TestBase classes 
    • should only contain state that all subclasses will need.
    • should be as small and clean as possible.
    • should not include utility methods, instead use utility classes. 
  • Unit instrumentation test classes should extend InstrumentationTestCase and not any other TestBase classes.
  • Utility test classes
    • are the preferred method of sharing functionality between test packages.
    • should only depend on production code or other test utilities.
    • should live in foo/test/android/javatests/src/bar/test/util.

Annotations

Each instrumentation test should have a test size and feature annotation.

Test Size Annotations

  • Always annotate a test with one of the test size annotations:
    • @SmallTest
    • @MediumTest
    • @LargeTest
    • android.test.suitebuilder.annotation.LargeTest/MediumTest/SmallTest 
  • Know the size of your test.
    • In general:
      • unit Instrumentation tests are small.
      • integration Instrumentation tests are medium or large.
  • The timeouts for each test size is:
    • @SmallTest - 1 minute
    • @MediumTest - 3 minutes
    • @LargeTest - 5 minutes

Feature Annotation

  • The feature annotation is used to specify which feature(s) the test is exercising. 
  • The list of features can be found in crbug.
  • To add a feature annotation to your test, add the @Feature annotation and specify the feature(s) that the test is testing. 
    • For example: @Feature({"Android-WebView"})
  • To run the tests for a specific feature, use:
    • python build/android/test_runner.py instrumentation --test-apk ContentShellTest --A Feature=Android-WebView,Main

Flaky and Crashing Tests

If a test is flaky or randomly crashes, the test should not run on the build bots.

The guidelines for handling flaky and crashing tests are as follows:
  • If a test is flaky but doesn’t crash, it should be marked flaky using the FlakyTest annotation.
  • If a test is crashing, it should be disabled using the DisabledTest annotation.
  • To mark a test flaky or disabled, do the following
    • create a bug for the test/area owner to fix the test.
    • don't mark a test flaky or disabled without creating a bug!!
    • add the @FlakyTest or @DisabledTest annotation to the test depending on whether the test is flaky or crashing.
    • add the bug number in a comment to the test.
    • please follow this example:
/*
@LargeTest
Bug XX
*/
@FlakyTest 
Replace @FlakyTest with @DisabledTest if disabling a test.
    • if the test has any other annotations, do NOT remove these! 
      • If the annotation is @Smoke, @SmallTest, @MediumTest, @LargeTest or @Feature, keep the annotation in the comment.
      • If the annotation is for example @PhoneOnly, keep the annotation as is.
Subpages (1): Android WebView tests
Comments