Runner

Application that runs a suite of tests.

Note: this application offers the full range of ScalaTest features via command line arguments described below. If you just want to run a suite of tests from the command line and see results on the standard output, you may prefer to use ScalaTest's simple runner.

The basic form of a Runner invocation is:

scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]

The arguments Runner accepts are described in the following table:

The simplest way to start Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:

scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -R compiled_tests

Given the previous command, Runner will discover and execute all Suites in the compiled_tests directory and its subdirectories, and show results in graphical user interface (GUI).

Executing suites

Each -s argument must be followed by one and only one fully qualified class name. The class must either extend Suite and have a public, no-arg constructor, or be annotated by a valid WrapWith annotation.

Specifying the config map

A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with "org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on the Runner command line. Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string. For example:

-Ddbname=testdb -Dserver=192.168.1.188

Specifying a runpath

A runpath is the list of filenames, directory paths, and/or URLs that Runner uses to load classes for the running test. If runpath is specified, Runner creates a custom class loader to load classes available on the runpath. The graphical user interface reloads the test classes anew for each run by creating and using a new instance of the custom class loader for each run. The classes that comprise the test may also be made available on the classpath, in which case no runpath need be specified.

The runpath is specified with the -R option. The -R must be followed by a space, a double quote ("), a white-space-separated list of paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off the double quotes, which only serve to combine a white-space separated list of strings into one command line argument. If you have path elements that themselves have a space in them, you must place a backslash (\) in front of the space. Here's an example:

-R "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"

Specifying reporters

Reporters can be specified on the command line in any of the following ways:

• -g[configs...] - causes display of a graphical user interface that allows tests to be run and results to be investigated
• -f[configs...] <filename> - causes test results to be written to the named file
• -u <directory> - causes test results to be written to junit-style xml files in the named directory
• -h <directory> [-Y <CSS file>] - causes test results to be written to HTML files in the named directory, optionally included the specified CSS file
• -a <number of files to archive> - causes specified number of old summary and durations files to be archived (in summaries/ and durations/ subdirectories) for dashboard reporter (default is two)
• -o[configs...] - causes test results to be written to the standard output
• -e[configs...] - causes test results to be written to the standard error
• -k <host> <port> - causes test results to be written to socket in the named host and port number, using XML format
• -K <host> <port> - causes test results to be written to socket in the named host and port number, using Java object binary format
• -C[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

The [configs...] parameter, which is used to configure reporters, is described in the next section.

The -C option causes the reporter specified in <reporterclass> to be instantiated. Each reporter class specified with a -C option must be public, implement org.scalatest.Reporter, and have a public no-arg constructor. Reporter classes must be specified with fully qualified names. The specified reporter classes may be deployed on the classpath. If a runpath is specified with the -R option, the specified reporter classes may also be loaded from the runpath. All specified reporter classes will be loaded and instantiated via their no-arg constructor.

For example, to run a suite named MySuite from the mydir directory using two reporters, the graphical reporter and a file reporter writing to a file named "test.out", you would type:

java -jar scalatest.jar -R mydir -g -f test.out -s MySuite

The -g, -o, or -e options can appear at most once each in any single command line. Multiple appearances of -f and -C result in multiple reporters unless the specified <filename> or <reporterclass> is repeated. If any of -g, -o, -e, <filename> or <reporterclass> are repeated on the command line, the Runner will print an error message and not run the tests.

Runner adds the reporters specified on the command line to a dispatch reporter, which will dispatch each method invocation to each contained reporter. Runner will pass the dispatch reporter to executed suites. As a result, every specified reporter will receive every report generated by the running suite of tests. If no reporters are specified, a graphical runner will be displayed that provides a graphical report of executed suites.

Configuring reporters

Each reporter option on the command line can include configuration characters. Configuration characters are specified immediately following the -g, -o, -e, -f, or -C. The following configuration characters, which cause reports to be dropped, are valid for any reporter:

• N - drop TestStarting events
• C - drop TestSucceeded events
• X - drop TestIgnored events
• E - drop TestPending events
• H - drop SuiteStarting events
• L - drop SuiteCompleted events
• O - drop InfoProvided events
• P - drop ScopeOpened events
• Q - drop ScopeClosed events
• R - drop ScopePending events
• M - drop MarkupProvided events

A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not present information about the event in its report. For example, if you specify -oN, the standard output reporter will never receive any TestStarting events and will therefore never report them. The purpose of these configuration parameters is to allow users to selectively remove events they find add clutter to the report without providing essential information.

The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:

• W - without color
• D - show all durations
• S - show short stack traces
• F - show full stack traces
• U - unformatted mode
• I - show reminder of failed and canceled tests without stack traces
• T - show reminder of failed and canceled tests with short stack traces
• G - show reminder of failed and canceled tests with full stack traces
• K - exclude TestCanceled events from reminder

If you specify a W, D, S, F, U, R, T, G, or K for any reporter other than standard output, standard error, or file reporters, Runner will complain with an error message and not perform the run.

Configuring a standard output, error, or file reporter with D will cause that reporter to print a duration for each test and suite. When running in the default mode, a duration will only be printed for the entire run.

Configuring a standard output, error, or file reporter with F will cause that reporter to print full stack traces for all exceptions, including TestFailedExceptions. Every TestFailedException contains a stack depth of the line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default, mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown by production code. When a TestFailedException is thrown in default mode, only the source filename and line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack trace.

The 'U' unformatted configuration removes some formatting from the output and adds verbosity. The purpose of unformatted (or, "ugly") mode is to facilitate debugging of parallel runs. If you have tests that fail or hang during parallel runs, but succeed when run sequentially, unformatted mode can help. In unformatted mode, you can see exactly what is happening when it is happening. Rather than attempting to make the output look as pretty and human-readable as possible, unformatted mode will just print out verbose information about each event as it arrives, helping you track down the problem you are trying to debug.

By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset terminal colors. Information printed as a result of run starting, completed, and stopped events is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green. The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode (W) will turn off this behavior. No ansi codes will be inserted.

The R, T, and G options enable "reminders" of failed and, optionally, canceled tests to be printed at the end of the summary. This minimizes or eliminates the need to search and scroll backwards to find out what tests failed or were canceled. For large test suites, the actual failure message could have scrolled off the top of the buffer, making it otherwise impossible to see what failed. You can configure the detail level of the stack trace for regular reports of failed and canceled tests independently from that of reminders. To set the detail level for regular reports, use S for short stack traces, F for full stack traces, or nothing for the default of no stack trace. To set the detail level for reminder reports, use T for reminders with short stack traces, G for reminders with full stack traces in reminders, or R for reminders with no stack traces. If you wish to exclude reminders of canceled tests, i.e., only see reminders of failed tests, specify K along with one of R, T, or G, as in "-oRK".

For example, to run a suite using two reporters, the graphical reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would type:

scala -classpath scalatest-<version>.jar -R mydir -g -eNDXEHLO -s MySuite

Note that no white space is allowed between the reporter option and the initial configuration parameters. So "-e NDXEHLO" will not work, "-eNDXEHLO" will work.

Specifying tags to include and exclude

You can specify tag names of tests to include or exclude from a run. To specify tags to include, use -n followed by a white-space-separated list of tag names to include, surrounded by double quotes. (The double quotes are not needed if specifying just one tag.) Similarly, to specify tags to exclude, use -l followed by a white-space-separated list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed if specifying just one tag.) If tags to include is not specified, then all tests except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed. (In other words, the absence of a -n option is like a wildcard, indicating all tests be included.) If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n and not mentioned in the tags to exclude, will be executed. For more information on test tags, see the documentation for Suite. Here are some examples:

• -n CheckinTests
• -n FunctionalTests -l org.scalatest.tags.Slow
• -n "CheckinTests FunctionalTests" -l "org.scalatest.tags.Slow org.scalatest.tags.Network"

Specifying suffixes to discover

You can specify suffixes of Suite names to discover. To specify suffixes to discover, use -q followed by a vertical-bar-separated list of suffixes to discover, surrounded by double quotes. (The double quotes are not needed if specifying just one suffix.) Or you can specify them individually using multiple -q's. If suffixes to discover is not specified, then all suffixes are considered. If suffixes is specified, then only those Suites whose class names end in one of the specified suffixes will be considered during discovery. Here are some examples:

• -q Spec
• -q "Spec|Suite"
• -q Spec -q Suite

Option -Q can be used to specify a default set of suffixes "Spec|Suite". If you specify both -Q and -q, you'll get Spec and Suite in addition to the other suffix or suffixes you specify with -q.

Specifying suffixes can speed up the discovery process because class files with names not ending the specified suffixes can be immediately disqualified, without needing to load and inspect them to see if they either extend Suite and declare a public, no-arg constructor, or are annotated with WrapWith.

Executing Suites in parallel

With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run tests in parallel. If you include -P on the command line, Runner will pass a Distributor to the Suites you specify with -s. Runner will set up a thread pool to execute any Suites passed to the Distributor's put method in parallel. Trait Suite's implementation of runNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suite of tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.

The -P option may optionally be appended with a number (e.g. "-P10" -- no intervening space) to specify the number of threads to be created in the thread pool. If no number (or 0) is specified, the number of threads will be decided based on the number of processors available.

Specifying Suites

Suites are specified on the command line with a -s followed by the fully qualified name of a Suite subclass, as in:

-s com.artima.serviceuitest.ServiceUITestkit

Each specified suite class must be public, a subclass of org.scalatest.Suite, and contain a public no-arg constructor. Suite classes must be specified with fully qualified names. The specified Suite classes may be loaded from the classpath. If a runpath is specified with the -R option, specified Suite classes may also be loaded from the runpath. All specified Suite classes will be loaded and instantiated via their no-arg constructor.

The runner will invoke execute on each instantiated org.scalatest.Suite, passing in the dispatch reporter to each execute method.

Runner is intended to be used from the command line. It is included in org.scalatest package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take over the role of runner, object org.scalatest.tools.Runner may be excluded from that implementation of the package. All other public types declared in package org.scalatest.tools.Runner should be included in any such usage, however, so client software can count on them being available.

Specifying "members-only" and "wildcard" Suite paths

If you specify Suite path names with -m or -w, Runner will automatically discover and execute accessible Suites in the runpath that are either a member of (in the case of -m) or enclosed by (in the case of -w) the specified path. As used in this context, a path is a portion of a fully qualified name. For example, the fully qualifed name com.example.webapp.MySuite contains paths com, com.example, and com.example.webapp. The fully qualifed name com.example.webapp.MyObject.NestedSuite contains paths com, com.example, com.example.webapp, and com.example.webapp.MyObject. An accessible Suite is a public class that extends org.scalatest.Suite and defines a public no-arg constructor. Note that Suites defined inside classes and traits do not have no-arg constructors, and therefore won't be discovered. Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus they can be discovered.

For example, if you specify -m com.example.webapp on the command line, and you've placed com.example.webapp.RedSuite and com.example.webapp.BlueSuite on the runpath, then Runner will instantiate and execute both of those Suites. The difference between -m and -w is that for -m, only Suites that are direct members of the named path will be discovered. For -w, any Suites whose fully qualified name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuite exists on the runpath, invoking Runner with -w com.example.webapp will cause GreenSuite to be discovered, because its fully qualifed name begins with "com.example.webapp". But if you invoke Runner with -m com.example.webapp, GreenSuite will not be discovered because it is directly a member of com.example.webapp.controllers, not com.example.webapp.

If you specify no -s, -m, or -w arguments on the command line to Runner, it will discover and execute all accessible Suites in the runpath.

Specifying chosen styles

You can optionally specify chosen styles for a ScalaTest run. ScalaTest supports different styles of testing so that different teams can use the style or styles that best suits their situation and culture. But in any one project, it is recommended you decide on one main style for unit testing, and consistently use only that style for unit testing throughout the project. If you also have integration tests in your project, you may wish to pick a different style for them than you are using for unit testing. You may want to allow certain styles to be used in special testing situations on a project, but in general, it is best to minimize the styles used in any given project to a few, or one.

To facilitate the communication and enforcement of a team's style choices for a project, you can specify the chosen styles in your project build. If chosen styles is defined, ScalaTest style traits that are not among the chosen list will abort with a message complaining that the style trait is not one of the chosen styles. The style name for each ScalaTest style trait is its fully qualified name. For example, to specify that org.scalatest.FunSpec as your chosen style you'd pass this to Runner:

-y org.scalatest.FunSpec

If you wanted org.scalatest.FunSpec as your main unit testing style, but also wanted to allow PropSpec for test matrixes and FeatureSpec for integration tests, you would write:

-y org.scalatest.FunSpec -y org.scalatest.PropSpec -y org.scalatest.FeatureSpec

To select org.scalatest.FlatSpec as your main unit testing style, but allow org.scalatest.fixture.FlatSpec for multi-threaded unit tests, you'd write:

-y org.scalatest.FlatSpec -y org.scalatest.fixture.FlatSpec

The style name for a suite is obtained by invoking its styleName method. Custom style traits can override this method so that a custom style can participate in the chosen styles list.

Because ScalaTest is so customizable, a determined programmer could circumvent the chosen styles check, but in practice -y should be persuasive enough tool to keep most team members in line.

Selecting suites and tests

Runner accepts three arguments that facilitate selecting suites and tests: -i, -t, and -z. The -i option enables a suite to be selected by suite ID. This argument is intended to allow tools such as IDEs or build tools to rerun specific tests or suites from information included in the results of a previous run. A -i must follow a -s that specifies a class with a public, no-arg constructor. The -i parameter can be used, for example, to rerun a nested suite that declares no zero-arg constructor, which was created by containing suite that does declare a no-arg constructor. In this case, -s would be used to specify the class ScalaTest can instantiate directly, the containing suite that has a public, no-arg constructor, and -i would be used to select the desired nested suite. One important use case for -i is to enable such a nested suite that aborted during the previous run to be rerun.

The -t argument allows a test to be selected by its (complete) test name. Like -i, the -t argument is primarily intented to be used by tools such as IDEs or build tools, to rerun selected tests based on information obtained from the results of a previous run. For example, -t could be used to rerun a test that failed in the previous run. The -t argument can be used directly by users, but because descriptive test names are usually rather long, the -z argument (described next), will usually be a more practical choice for users. If a -t follows either -s or -i, then it only applies to the suite identified. If it is specified independent of a -s or -i, then discovery is performed to find all Suites containing the test name.

The -z option allows tests to be selected by a simplified wildcard: any test whose name includes the substring specified after -z will be selected. For example, -z popped would select tests named "An empty stack should complain when popped" and "A non-empty stack should return the last-pushed value when popped, but not "An empty stack should be empty". In short, -z popped would select any tests whose name includes the substring "popped", and not select any tests whose names don't include "popped". This simplified approach to test name wildcards, which was suggested by Mathias Doenitz, works around the difficulty of finding an actual wildcard character that will work reliably on different operating systems. Like -t, if -z follows -s or -i, then it only applies to the Suite specified. Otherwise discovery is performed to find all Suites containing test names that include the substring.

Specifying a span scale factor

If you specify a integer or floating point span scale factor with -F, trait ScaledTimeSpans trait will return the specified value from its implementation of spanScaleFactor. This allows you to tune the "patience" of a run (how long to wait for asynchronous operations) from the command line. For more information, see the documentation for trait ScaledTimeSpans.

Specifying TestNG XML config file paths

If you specify one or more file paths with -b (b for Beust, the last name of TestNG's creator), Runner will create a org.scalatest.testng.TestNGWrapperSuite, passing in a List of the specified paths. When executed, the TestNGWrapperSuite will create one TestNG instance and pass each specified file path to it for running. If you include -b arguments, you must include TestNG's jar file on the class path or runpath. The -b argument will enable you to run existing TestNG tests, including tests written in Java, as part of a ScalaTest run. You need not use -b to run suites written in Scala that extend TestNGSuite. You can simply run such suites with -s, -m, or -w parameters.

Specifying JUnit tests

JUnit tests, including ones written in Java, may be run by specifying -j classname, where the classname is a valid JUnit class such as a TestCase, TestSuite, or a class implementing a static suite() method returning a TestSuite.

To use this option you must include a JUnit jar file on your classpath.

Memorizing and rerunning failed and canceled tests

You can memorize failed and canceled tests using -M:

-M failed-canceled.txt

All failed and canceled tests will be memorized in failed-canceled.txt, to rerun them again, you use -A:

-A failed-canceled.txt

Slowpoke notifications

You can request to recieve periodic notifications of slowpokes, tests that have been running longer than a given amount of time, specified in seconds by the first integer after -W, the delay. You specify the period between slowpoke notifications in seconds with the second integer after -W, the period. Thus to receive notifications very minute of tests that have been running longer than two minutes, you'd use:

-W 120 60

Slowpoke notifications will be sent via AlertProvided events. The standard out reporter, for example, will report such notifications like:

*** Test still running after 2 minutes, 13 seconds: suite name: ExampleSpec, test name: An egg timer should take 10 minutes.