Keywords: This testing document describes how to run GLBench, TearTest, Piglit, WebGLConformance and other tests and benchmarks on ChromeOS and ChromiumOS devices. Source code locationPython sources are under src/third_party/autotest/files/client/site_tests
Binary dependencies are under src/third_party/autotest/files/client/deps
A note on building binariesThe ebuild system knows if a target knows either OpenGL or OpenGL ES, or even both. If only pre-compiled binaries are used there should not be much of a problem with running tests. When editing tests after using the cros_workon script the default system behavior is to recompile the binaries once more without invoking the ebuild system. Unfortunately this means USE="opengles" flags are not honored. Instead the easy way is to specify the proper switch via am environment variable GRAPHICS_BACKEND=OPENGLES./run_remote_tests.sh that gets directly passed to the Makefile. This is somewhat hacky. To use the ebuild system to generate the binaries one has to do a painfully slow USE=opengles emerge-tegra2_seaboard chromeos-base/autotest followed by ./run_remote_tests.sh --use_emerged [...]. Don't forget the --use_emerged flag as it is not default. Autotest graphics_GLAPICheck- two different binaries can be made -- GRAPHICS_BACKEND=OPENGL or OPENGLES - runs 25 seconds - sample results - for OPENGL creates output below and parses some samples in python to see if they are there and if SUCCEED: run to the end is there GL_VERSION = 3.2.0 NVIDIA 195.36.24 GL_EXTENSIONS = GL_ARB_color_buffer_float GL_ARB_compatibility GL_ARB_copy_buffer GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex [...] NV-GLX NV-CONTROL Generic Event Extension SHAPE MIT-SHM XInputExtension XTEST BIG-REQUESTS SYNC XKEYBOARD XC-MISC SECURITY XINERAMA XFIXES RENDER RANDR XINERAMA Composite DAMAGE SUCCEED: run to the end Sample output(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.71.194 graphics_GLAPICheck --use_emerged Initiating first contact with remote host [...] INFO : Test results: --------------------------------------------- graphics_GLAPICheck PASS graphics_GLAPICheck/graphics_GLAPICheck PASS --------------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 0m25s Autotest graphics_GLBench- draws green/ref/purple/fractal rectangles - runs about 5 minutes This benchmark executes glbench, a graphics benchmark designed to time how long various graphic intensive activities take, which includes measuring: - fill rate - blended - opaque -Z reject rate -triangle rate - no cull - half cull (half triangles backface culled) - full cull (mix of back face and degenerates) - blend rate - texture fetch - nearest - bilinear - trilinear - compute - vertex shader - pixel shader - *fragement shader to test ddx and ddy - attribute fetch - color depth stencil test - *state change - texture upload - read back - does MD5 checksums of some but not all images -- deps/glbench/src/checksums -- can be re-generated with -save option Sample output(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGLES ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.27 graphics_GLBench --use_emerged [...] INFO : Test results: ---------------------------------------------------- graphics_GLBench PASS graphics_GLBench/graphics_GLBench PASS 1280x768_fps_compositing 11.18 1280x768_fps_no_fill_compositing 41.96 mpixels_sec_clear_color 490.74 mpixels_sec_clear_colordepth 323.88 mpixels_sec_clear_colordepthstencil 281.45 mpixels_sec_clear_depth 969.71 mpixels_sec_clear_depthstencil 647.34 mpixels_sec_fill_solid 285.1 mpixels_sec_fill_solid_blended 128.55 mpixels_sec_fill_solid_depth_neq 973.2 mpixels_sec_fill_solid_depth_never 1189.08 mpixels_sec_fill_tex_bilinear 149.56 mpixels_sec_fill_tex_nearest 203.7 mpixels_sec_fill_tex_trilinear_linear_01 164.63 mpixels_sec_fill_tex_trilinear_linear_04 nan mpixels_sec_fill_tex_trilinear_linear_05 nan mpixels_sec_fill_tex_trilinear_nearest_05 nan mpixels_sec_pixel_read 6.72 mpixels_sec_pixel_read_2 6.72 mpixels_sec_pixel_read_3 6.74 mpixels_sec_varyings_shader_1 270.68 mpixels_sec_varyings_shader_2 144.33 mpixels_sec_varyings_shader_4 73.57 mpixels_sec_varyings_shader_8 24.29 mpixels_sec_yuv_shader_1 nan mpixels_sec_yuv_shader_2 47.82 mpixels_sec_yuv_shader_3 93.36 mpixels_sec_yuv_shader_4 136.15 mtexel_sec_texture_update_teximage2d_1024 149.85 mtexel_sec_texture_update_teximage2d_128 80.92 mtexel_sec_texture_update_teximage2d_1536 146.05 mtexel_sec_texture_update_teximage2d_2048 149.47 mtexel_sec_texture_update_teximage2d_256 132.51 mtexel_sec_texture_update_teximage2d_32 10.29 mtexel_sec_texture_update_teximage2d_512 142.05 mtexel_sec_texture_update_teximage2d_768 147.56 mtexel_sec_texture_update_texsubimage2d_1024 150.94 mtexel_sec_texture_update_texsubimage2d_128 81.59 mtexel_sec_texture_update_texsubimage2d_1536 147.2 mtexel_sec_texture_update_texsubimage2d_2048 149.03 mtexel_sec_texture_update_texsubimage2d_256 130.59 mtexel_sec_texture_update_texsubimage2d_32 10.09 mtexel_sec_texture_update_texsubimage2d_512 113.54 mtexel_sec_texture_update_texsubimage2d_768 150.0 mtri_sec_triangle_setup 30.34 mtri_sec_triangle_setup_all_culled 49.89 mtri_sec_triangle_setup_half_culled 44.91 mvtx_sec_attribute_fetch_shader 149.51 mvtx_sec_attribute_fetch_shader_2_attr 149.34 mvtx_sec_attribute_fetch_shader_4_attr 141.36 mvtx_sec_attribute_fetch_shader_8_attr 106.71 us_swap_swap 19464.19 ---------------------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 8m58s >>> Details stored under /tmp/run_remote_tests.purz The "nan" in this example are passed because of knownbad output images. For more more details and failures look at (cros-chroot) ihf@ql ~/trunk/src/scripts $ more /tmp/run_remote_tests.purz/graphics_GLBench/graphics_GLBench/summary.txt Note on HW Qual failuresSometimes when site_tests/suite_HWQual/control.auto is run a constraint is violated and failure mtri_sec_triangle_setup >= 10 is thrown. This is due to some x86 boards doing vertex shading in the CPU. This exception is considered harmless until a waiver system is developed. Autotest graphics_O3DSelenium- tries to run O3D tests inside of chrome - failed to run and is known to be broken Autotest graphics_Piglit- tests OpenGL implementation. Does not currently work on OpenGL ES only systems like Tegra2 - about 1000-2000 targeted subtests - usually a few hundred failures - mostly care about counts of passes/failures over time like perf test - runs about 12-18 minutes, but initial copying dep-piglit.tar.bz2 takes 5-10 minutes that seem like a hang. Need to fix this. - generates html page with detailed results in /tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html - can compare two test runs against each other via ./piglit-summary-html.py summary/compare results/baseline.results results/current.results - can be build locally from piglit.tar.gz via cmake .;make - the numbers of passes does fluctuate by about 10 over time, so some intermittency - about 10 test binaries crash and are blacklisted to not be reported - sometimes (1 in about 2 weeks of test runs) fbo_depth_sample_compare takes gpu/system with it when it crashes 2011-05-04T11:06:02.995237-07:00 localhost kernel: [ 75.994363] fbo-depth-sampl[4250]: segfault at 0 ip (null) sp 7fc7abac error 14 in card0[75f97000+4000] 2011-05-04T11:06:03.040387-07:00 localhost crash_reporter[4251]: Received crash notification for fbo-depth-sample-compare[4250] sig 11 (developer build - not testing - always dumping) 2011-05-04T11:06:03.234936-07:00 localhost crash_reporter[4251]: Stored minidump to /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.dmp 2011-05-04T11:06:03.235566-07:00 localhost crash_reporter[4251]: Leaving core file at /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.core due to developer image 2011-05-04T11:06:17.742377-07:00 localhost kernel: [ 90.742080] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung 2011-05-04T11:06:17.742439-07:00 localhost kernel: [ 90.742100] render error detected, EIR: 0x00000000 2011-05-04T11:06:17.742471-07:00 localhost kernel: [ 90.742168] [drm:i915_do_wait_request] *ERROR* i915_do_wait_request returns -5 (awaiting 29946 at 29941) 2011-05-04T11:06:18.330692-07:00 localhost crash_reporter[4328]: Received crash notification for Xorg[3498] sig 6 (developer build - not testing - always dumping) Sample output(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_Piglit --use_emerged [...] INFO : Test results: ------------------------------------- graphics_Piglit PASS graphics_Piglit/graphics_Piglit PASS count_subtests_fail 239 count_subtests_pass 1040 count_subtests_skip 687 count_subtests_warn 14 ------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 13m31s >>> Details stored under /tmp/run_remote_tests.8gjT Point browser to /tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html Autotest graphics_SanAngeles- draws a bunch of very coarse 3d models This test runs the San Angeles Observation GPU benchmark. This benchmark uses a minimal and portable framework to generate a small demo program. It exercises basic features of OpenGL like vertex arrays, color arrays, and lighting. It also uses objects generated using procedural algorithms. This test is a benchmark. It will fail if it fails to complete. Sample output(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_SanAngeles --use_emerged [...] INFO : Test results: --------------------------------------------- graphics_SanAngeles PASS graphics_SanAngeles/graphics_SanAngeles PASS frames_per_sec_rate_san_angeles 2.1 --------------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 2m17s >>> Details stored under /tmp/run_remote_tests.vDwN Autotest graphics_TearTest- copies glbench as dependency over - runs deps/glbench/teartest - This test will fail if there is tearing in the two vertical lines that are scrolling horizontally. This is a semi-automated test that displays vertical lines scrolling horizontally and asks the user if tearing was observed. Three variants are available: * using uniform update. This tests that glSwapInterval function performs as expected. * using full texture update. This tests that CPU-GPU interaction is properly synchronized in the driver. * using pixmap to texture extension. This tests that pixmap to texture extension is properly synchronized. (cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_TearTest --use_emerged [...] - some user interaction INFO : Test results: ----------------------------------------- graphics_TearTest PASS graphics_TearTest/graphics_TearTest PASS ----------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 1m26s >>> Details stored under /tmp/run_remote_tests.SHxS Autotest graphics_WebGLConformanceThis test runs the WebGL conformance tests: http://www.khronos.org/webgl/wiki/Testing/Conformance - currently uses release version 1.0.0 - opens copy of above website in chrome and executes the html page with hundreds of targeted tests - all suite tests must pass, but there is the ability to waive failures in the python script - javascript is patched to report individual results - suite is currently blocked in autotest regression runs by hangs/crashes in browser Sample outputDoesn't compile any code, so no GRAPHICS_BACKEND needed (but no harm in using it). When the browser doesn't crash/hang it should look like this. (cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.80 graphics_WebGLConformance/control [...] INFO : Test results: --------------------------------------------------------- graphics_WebGLConformance PASS graphics_WebGLConformance/graphics_WebGLConformance PASS count_tests_fail 352 count_tests_pass 5064 count_tests_timeout 2 waived_url_000 conformance/tex-image-and-sub-image-2d-with-array-buffer-view.html : 192 failures (192 waived) waived_url_001 conformance/tex-image-with-format-and-type.html : 12 failures (12 waived) waived_url_002 conformance/texture-npot.html : 12 failures (12 waived) waived_url_003 conformance/glsl-conformance.html : 1 failures (1 waived) waived_url_004 conformance/tex-image-and-sub-image-2d-with-image.html : 8 failures (8 waived) waived_url_005 conformance/copy-tex-image-and-sub-image-2d.html : 34 failures (34 waived) waived_url_006 conformance/gl-clear.html : 4 failures (4 waived) waived_url_007 conformance/more/functions/readPixelsBadArgs.html : 1 failures (1 waived) waived_url_008 conformance/more/conformance/webGLArrays.html : 1 failures (1 waived) waived_url_009 conformance/gl-teximage.html : 46 failures (46 waived) waived_url_010 conformance/texture-active-bind.html : 4 failures (4 waived) waived_url_011 conformance/read-pixels-test.html : 3 failures (3 waived) waived_url_012 conformance/gl-object-get-calls.html : 2 failures (2 waived) waived_url_013 conformance/point-size.html : 1 failures (1 waived) waived_url_014 conformance/texture-formats-test.html : 4 failures (4 waived) waived_url_015 conformance/texture-complete.html : 1 failures (1 waived) waived_url_016 conformance/tex-image-and-sub-image-2d-with-video.html : 8 failures (8 waived) waived_url_017 conformance/context-lost-restored.html : 2 failures (2 waived) waived_url_018 conformance/tex-image-and-sub-image-2d-with-image-data.html : 16 failures (16 waived) --------------------------------------------------------- Total PASS: 2/2 (100%) Crashes detected during testing: --------------------------------------------------------- chrome sig 6 graphics_WebGLConformance/graphics_WebGLConformance --------------------------------------------------------- Total unique crashes: 1 >>> Details stored under /tmp/run_remote_tests.iumX Autotest graphics_WindowManagerGraphicsCapture- This test verifies the window manager can capture graphics from applications. - This test fails if application screen shots cannot capture the screen output. - draws less than 10 full screen color frames of a dot/sphere - copies dep-glbench.tar.bz2 as dependency over - runs deps/glbench/windowmanagertest - converts/resizes images to 100x100 pixels - runs perceptualdiff to compare with reference images - runs about 30 seconds - fairly stable test Sample Outputihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_WindowManagerGraphicsCapture --use_emerged [...] INFO : Test results: --------------------------------------------------------------------------------- graphics_WindowManagerGraphicsCapture PASS graphics_WindowManagerGraphicsCapture/graphics_WindowManagerGraphicsCapture PASS --------------------------------------------------------------------------------- Total PASS: 2/2 (100%) No crashes detected during testing. Elapsed time: 0m36s >>> Details stored under /tmp/run_remote_tests.Lb4p |
Chromium OS > Testing Home >