174e88debSCassandra TargettTesting 274e88debSCassandra Targett======= 374e88debSCassandra Targett 474e88debSCassandra TargettExamples below assume cwd at the gradlew script in the top directory of 574e88debSCassandra Targettthe project's checkout. 674e88debSCassandra Targett 774e88debSCassandra Targett 874e88debSCassandra TargettGeneric test/ checkup commands 974e88debSCassandra Targett------------------------------ 1074e88debSCassandra Targett 1174e88debSCassandra TargettRun all unit tests: 1274e88debSCassandra Targett 1374e88debSCassandra Targettgradlew test 1474e88debSCassandra Targett 1574e88debSCassandra TargettRun all verification tasks, including tests: 1674e88debSCassandra Targett 1774e88debSCassandra Targettgradlew check 1874e88debSCassandra Targett 1974e88debSCassandra TargettRun all verification tasks, excluding tests (-x is gradle's generic task 2074e88debSCassandra Targettexclusion mechanism): 2174e88debSCassandra Targett 2274e88debSCassandra Targettgradlew check -x test 2374e88debSCassandra Targett 2474e88debSCassandra TargettRun verification for a selected module only: 2574e88debSCassandra Targett 2674e88debSCassandra Targettgradlew :lucene:core:check # By full gradle project path 2774e88debSCassandra Targettgradlew -p lucene/core check # By folder designation + task 2874e88debSCassandra Targett 2974e88debSCassandra Targett 3074e88debSCassandra TargettRandomization 3174e88debSCassandra Targett------------- 3274e88debSCassandra Targett 3374e88debSCassandra TargettTo run tests with the given starting seed pass 'tests.seed' 3474e88debSCassandra Targettproperty: 3574e88debSCassandra Targett 3674e88debSCassandra Targettgradlew :lucene:misc:test -Ptests.seed=DEADBEEF 3774e88debSCassandra Targett 3874e88debSCassandra TargettThere are a lot of other test randomization properties 3974e88debSCassandra Targettavailable. To list them, their defaults and current values 4074e88debSCassandra Targettrun the testOpts task against a project that has tests. 4174e88debSCassandra TargettFor example: 4274e88debSCassandra Targett 4374e88debSCassandra Targettgradlew -p lucene/core testOpts 4474e88debSCassandra Targett 4574e88debSCassandra Targett 4674e88debSCassandra TargettFiltering 4774e88debSCassandra Targett--------- 4874e88debSCassandra Targett 4974e88debSCassandra TargettRun tests of lucene-core module: 5074e88debSCassandra Targett 5174e88debSCassandra Targettgradlew -p lucene/core test 5274e88debSCassandra Targett 5374e88debSCassandra TargettRun a single test case (from a single module). Uses gradle's built-in filtering 5474e88debSCassandra Targett(https://docs.gradle.org/current/userguide/java_testing.html#test_filtering): 5574e88debSCassandra Targett 5674e88debSCassandra Targettgradlew -p lucene/core test --tests TestDemo 5774e88debSCassandra Targett 5874e88debSCassandra TargettRun all tests in a package: 5974e88debSCassandra Targett 6074e88debSCassandra Targettgradlew -p lucene/core test --tests "org.apache.lucene.document.*" 6174e88debSCassandra Targett 6274e88debSCassandra TargettRun all test classes/ methods that match this pattern: 6374e88debSCassandra Targett 6474e88debSCassandra Targettgradlew -p lucene/core test --tests "*testFeatureMissing*" 6574e88debSCassandra Targett 6674e88debSCassandra Targett 6774e88debSCassandra TargettTest groups 6874e88debSCassandra Targett----------- 6974e88debSCassandra Targett 7074e88debSCassandra TargettTests can be filtered by an annotation they're marked with. 71*3edfeb5eSRobert MuirSome test group annotations include: @AwaitsFix, @Nightly 7274e88debSCassandra Targett 7374e88debSCassandra TargettThis uses filtering infrastructure on the *runner* (randomizedtesting), 7474e88debSCassandra Targettnot gradle's built-in mechanisms (but it can be combined with "--tests"). 75*3edfeb5eSRobert MuirFor example, run all lucene-core tests annotated as @Nightly: 7674e88debSCassandra Targett 77*3edfeb5eSRobert Muirgradlew -p lucene/core test -Ptests.filter=@Nightly 7874e88debSCassandra Targett 7974e88debSCassandra TargettTest group filters can be combined into Boolean expressions: 8074e88debSCassandra Targett 81*3edfeb5eSRobert Muirgradlew -p lucene/core test "default and not(@awaitsfix or @nightly)" 8274e88debSCassandra Targett 8374e88debSCassandra Targett 8474e88debSCassandra TargettReiteration ("beasting") 8574e88debSCassandra Targett------------------------ 8674e88debSCassandra Targett 8774e88debSCassandra TargettMultiply each test case N times (this works by repeating the same test 8874e88debSCassandra Targettwithin the same JVM; it also works in IDEs): 8974e88debSCassandra Targett 9074e88debSCassandra Targettgradlew -p lucene/core test --tests TestDemo -Ptests.iters=5 9174e88debSCassandra Targett 9274e88debSCassandra TargettTests tasks will be (by default) re-executed on each invocation because 93486141f0SDawid Weisswe pick a random global tests.seed and because we want them to (force 94486141f0SDawid Weissthe test task to run). If you want to make the 'tests' task obey up-to-date 95486141f0SDawid Weissgradle rules, fix the seed and turn off up to date trigger: 9674e88debSCassandra Targett 97486141f0SDawid Weissgradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.neverUpToDate=false 9874e88debSCassandra Targett 99486141f0SDawid WeissThese properties can be set in your local gradle.properties. To force re-execution 100486141f0SDawid Weissof tests, even for the same master seed, apply cleanTest task: 10174e88debSCassandra Targett 10274e88debSCassandra Targettgradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef 10374e88debSCassandra Targett 10483ed210fSDawid WeissThe 'tests.iters' option should be sufficient for individual test cases 10583ed210fSDawid Weissand is *much* faster than trying to duplicate re-runs of the entire 10683ed210fSDawid Weisstest suites. When it is absolutely needed to re-run an entire suite (because 10783ed210fSDawid Weissof randomization in the static initialization, for example), you can do it 10883ed210fSDawid Weissby running the 'beast' task with 'tests.dups' option: 10983ed210fSDawid Weiss 11083ed210fSDawid Weissgradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat 11183ed210fSDawid Weiss 11283ed210fSDawid WeissNote the filter (--tests) used to narrow down test reiterations to a particular 11383ed210fSDawid Weissclass. You can use any filter, including no filter at all, but it rarely makes 11483ed210fSDawid Weisssense (will take ages). By default the test tasks generated by the 'beast' mode 11583ed210fSDawid Weissuse a random starting seed for randomization. If you pass an explicit seed, this 11683ed210fSDawid Weisswon't be the case (all tasks will use exactly the same starting seed): 11783ed210fSDawid Weiss 11883ed210fSDawid Weissgradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef 11974e88debSCassandra Targett 12006907a2cSRobert Muir 12174e88debSCassandra TargettVerbose mode and debugging 12274e88debSCassandra Targett-------------------------- 12374e88debSCassandra Targett 12474e88debSCassandra TargettThe "tests.verbose" mode switch enables standard streams from tests 12574e88debSCassandra Targettto be dumped directly to the console. Run your verbose tests explicitly 12674e88debSCassandra Targettspecifying the project and test task or a fully qualified task path. Example: 12774e88debSCassandra Targett 12874e88debSCassandra Targettgradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo" 12974e88debSCassandra Targett 13074e88debSCassandra Targett 1311089b482SRobert MuirRun GUI tests headlessly with Xvfb (Linux only) 1321089b482SRobert Muir----------------------------------------------- 1331089b482SRobert Muir 1341089b482SRobert MuirGUI test for Luke application launches a window, this might mess up your 1351089b482SRobert Muirmonitor depending on the display manager you are using. In that case, 1361089b482SRobert Muiryou can install Xvfb (X Virtual Frame Buffer) so that the test runs on the 1371089b482SRobert Muirvirtual display and does not open a real window. 1381089b482SRobert Muir 1391089b482SRobert Muir# redhat-type OS 1401089b482SRobert Muir$ sudo yum install Xvfb 1411089b482SRobert Muir 1421089b482SRobert Muir# ubuntu/debian-type OS 1431089b482SRobert Muir$ sudo apt install xvfb 1441089b482SRobert Muir 1451089b482SRobert Muir# arch 1461089b482SRobert Muir$ sudo pacman -S xorg-server-xvfb 1471089b482SRobert Muir 1481089b482SRobert Muir 149e504798aSRobert MuirProfiling slow tests 150e504798aSRobert Muir-------------------- 151e504798aSRobert Muir 152e504798aSRobert MuirThe "tests.profile" mode switch turns on a sampling profiler during test execution, 1534b5105e1SRobert Muirand prints a simple summary at the end. 1544b5105e1SRobert Muir 1554b5105e1SRobert MuirFor example, top-10 histogram of methods (cpu samples): 156e504798aSRobert Muir 157e504798aSRobert Muirgradlew -p lucene/core test -Ptests.profile=true 158e504798aSRobert Muir 1594b5105e1SRobert MuirAlternatively, you can profile heap allocations instead: 1604b5105e1SRobert Muir 1614b5105e1SRobert Muirgradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap 1624b5105e1SRobert Muir 1634b5105e1SRobert MuirBy default, results are computed (deduplicated) on just the method name, folding 1644b5105e1SRobert Muirtogether all events from the same method. To drill down further, you can increase the 1654b5105e1SRobert Muirstack size from the default of 1, to get a histogram of stacktraces instead: 1664b5105e1SRobert Muir 1674b5105e1SRobert Muirgradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8 1684b5105e1SRobert Muir 1694b5105e1SRobert MuirFor big methods, it can also be helpful to include line numbers for more granularity: 1704b5105e1SRobert Muir 1714b5105e1SRobert Muirgradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true 1724b5105e1SRobert Muir 1734b5105e1SRobert MuirUsing these additional options will make the results more sparse, so it may be useful 1744b5105e1SRobert Muirto increase the top-N count: 1754b5105e1SRobert Muir 176ed7f507cSRobert Muirgradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100 177246c4bebSDawid Weiss 178246c4bebSDawid Weiss 17906907a2cSRobert MuirGenerating Coverage Reports 18006907a2cSRobert Muir------------------------------ 18106907a2cSRobert Muir 18206907a2cSRobert MuirRunning the "coverage" task (or setting the property "tests.coverage" to true) 18306907a2cSRobert Muirwill run the tests with instrumentation to record code coverage. 18406907a2cSRobert Muir 18506907a2cSRobert MuirExample: 18606907a2cSRobert Muir 18706907a2cSRobert Muirgradlew -p lucene/core coverage 18806907a2cSRobert Muiropen lucene/core/build/reports/jacoco/test/html/index.html 18906907a2cSRobert Muir 1909a17d676SNhat NguyenIf you want to use test filtering to just check a particular test, specify 1919a17d676SNhat Nguyenthe "test" task explicitly before "coverage": 1929a17d676SNhat Nguyen 1939a17d676SNhat Nguyengradlew -p lucene/core test --tests TestDemo coverage 1949a17d676SNhat Nguyenopen lucene/core/build/reports/jacoco/test/html/index.html 1959a17d676SNhat Nguyen 19606907a2cSRobert Muir 197246c4bebSDawid WeissExternal data sets 198246c4bebSDawid Weiss------------------ 199246c4bebSDawid Weiss 200246c4bebSDawid WeissSome tests may require external (and large) data sets. To see relevant tasks 201246c4bebSDawid Weissthat download and extract these data files automatically, run the following: 202246c4bebSDawid Weiss 203246c4bebSDawid Weissgradlew tasks --group "Data set download" 204