Application that runs a suite of tests.
The application accepts command line arguments that specify optional config map (key-value pairs), an optional
runpath, zero to many
Reporters, optional lists of tags to include and/or exclude, zero to many
Suite class names, zero to many "members-only"
Suite paths, zero to many "wildcard"
and zero to many TestNG XML config file paths.
All of these arguments are described in more detail below. Here's a summary:
scala [-classpath scalatest-<version>.jar:...] org.scalatest.tools.Runner [-D<key>=<value> [...]] [-p <runpath>] [reporter [...]] [-n <includes>] [-l <excludes>] [-c] [-s <suite class name> [...]] [-j <junit class name> [...]] [-m <members-only suite path> [...]] [-w <wildcard suite path> [...]] [-t <TestNG config file path> [...]]
The simplest way to start
Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:
scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -p compiled_tests
Given the previous command,
Runner will discover and execute all
Suites in the
compiled_tests directory and its subdirectories,
and show results in graphical user interface (GUI).
Specifying the config map
A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with
"org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on the
Runner command line.
Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string.
Specifying a runpath
A runpath is the list of filenames, directory paths, and/or URLs that
uses to load classes for the running test. If runpath is specified,
a custom class loader to load classes available on the runpath.
The graphical user interface reloads the test classes anew for each run
by creating and using a new instance of the custom class loader for each run.
The classes that comprise the test may also be made available on
the classpath, in which case no runpath need be specified.
The runpath is specified with the -p option. The -p must be followed by a space,
a double quote (
"), a white-space-separated list of
paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off
the double quotes, which only serve to combine a white-space separated list of strings into one
command line argument. If you have path elements that themselves have a space in them, you must
place a backslash (\) in front of the space. Here's an example:
-p "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"
Reporters can be specified on the command line in any of the following ways:
-g[configs...]- causes display of a graphical user interface that allows tests to be run and results to be investigated
-f[configs...] <filename>- causes test results to be written to the named file
-u <directory>- causes test results to be written to xml files in the named directory
-o[configs...]- causes test results to be written to the standard output
-e[configs...]- causes test results to be written to the standard error
-r[configs...] <reporterclass>- causes test results to be reported to an instance of the specified fully qualified
[configs...] parameter, which is used to configure reporters, is described in the next section.
-r option causes the reporter specified in
<reporterclass> to be
Each reporter class specified with a -r option must be public, implement
org.scalatest.Reporter, and have a public no-arg constructor.
Reporter classes must be specified with fully qualified names.
The specified reporter classes may be
deployed on the classpath. If a runpath is specified with the
-p option, the specified reporter classes may also be loaded from the runpath.
All specified reporter classes will be loaded and instantiated via their no-arg constructor.
For example, to run a suite named
MySuite from the
using two reporters, the graphical reporter and a file reporter
writing to a file named
"test.out", you would type:
java -jar scalatest.jar -p mydir -g -f test.out -s MySuite
-e options can
appear at most once each in any single command line.
Multiple appearances of
-r result in multiple reporters
unless the specified
repeated. If any of
<reporterclass> are repeated on
the command line, the
Runner will print an error message and not run the tests.
Runner adds the reporters specified on the command line to a dispatch reporter,
which will dispatch each method invocation to each contained reporter.
Runner will pass
the dispatch reporter to executed suites. As a result, every
specified reporter will receive every report generated by the running suite of tests.
If no reporters are specified, a graphical
runner will be displayed that provides a graphical report of
Each reporter option on the command line can include configuration characters. Configuration characters
are specified immediately following the
-r. The following configuration
characters, which cause reports to be dropped, are valid for any reporter:
A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not
present information about the event in its report. For example, if you specify
-oN, the standard output reporter
will never receive any
TestStarting events and will therefore never report them. The purpose of these
configuration parameters is to allow users to selectively remove events they find add clutter to the report without
providing essential information.
Deprecation Note: Prior to 1.0, ScalaTest's
Runner allowed you specify configuration parameters on reports that
indicated a particular event should be presented. This meant that people could opt to not show
test failures, suite aborted events, etc. To prevent important events from being dropped accidentally,
starting in 1.0 the configuration parameters indicate which events should not be presented, and important
events can't be dropped at all. For two releases,
the old config parameters will be tolerated, but have no effect (except for F, which turns on printing of
stack traces). Only the new parameters will have any effect,
and none of the new ones overlap with any of the old ones. So you have two releases to change your scripts to
use the new config parameters. Starting with 1.2, using the old parameters—Y, Z, T, F, G, U, P, B, I, S, A, R—will
Runner to abort with an error message and not run the tests.
The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:
W- without color
D- show all durations
If you specify a W, D, or F for any reporter other than standard output, standard error, or file reporters,
will complain with an error message and not perform the run.
Configuring a standard output, error, or file reporter with
D will cause that reporter to
print a duration for each test and suite. When running in the default mode, a duration will only be printed for
the entire run.
Configuring a standard output, error, or file reporter with
F will cause that reporter to print full stack traces for all exceptions,
TestFailedException contains a stack depth of the
line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default,
mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown
by production code. When a
TestFailedException is thrown in default mode, only the source filename and
line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack
By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset terminal colors. Information printed as a result of run starting, completed, and stopped events is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green. The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode ('W') will turn off this behavior. No ansi codes will be inserted.
For example, to run a suite using two reporters, the graphical reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would type:
scala -classpath scalatest-<version>.jar -p mydir -g -eNDXEHLO -s MySuite
Note that no white space is allowed between the reporter option and the initial configuration
"-e NDXEHLO" will not work,
"-eNDXEHLO" will work.
Specifying tags to include and exclude
You can specify tag names of tests to include or exclude from a run. To specify tags to include,
-n followed by a white-space-separated list of tag names to include, surrounded by
double quotes. (The double quotes are not needed if specifying just one tag.) Similarly, to specify tags
to exclude, use
-l followed by a white-space-separated
list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed
if specifying just one tag.) If tags to include is not specified, then all tests
except those mentioned in the tags to exclude (and in the
org.scalatest.Ignore tag), will be executed.
(In other words, the absence of a
-n option is like a wildcard, indicating all tests be included.)
If tags to include is specified, then only those tests whose tags are mentioned in the argument following
and not mentioned in the tags to exclude, will be executed. For more information on test tags, see
the documentation for
Suite. Here are some examples:
-n FunctionalTests -l SlowTests
-n "CheckinTests FunctionalTests" -l "SlowTests NetworkTests"
With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run
tests concurrently. If you include
-c on the command line,
Runner will pass a
Suites you specify with
Runner will set up a thread pool to execute any
passed to the
put method concurrently. Trait
Suite's implementation of
runNestedSuites will place any nested
Suites into this
Distributor. Thus, if you have a
of tests that must be executed sequentially, you should override
runNestedSuites as described in the documentation for
-c option may optionally be appended with a number (e.g.
-c10" -- no intervening space) to specify the number of
threads to be created in the thread pool. If no number (or 0) is
specified, the number of threads will be decided based on the number of
Suites are specified on the command line with a -s followed by the fully qualified
name of a
Suite subclass, as in:
Each specified suite class must be public, a subclass of
org.scalatest.Suite, and contain a public no-arg constructor.
Suite classes must be specified with fully qualified names.
Suite classes may be
loaded from the classpath. If a runpath is specified with the
-p option, specified
Suite classes may also be loaded from the runpath.
Suite classes will be loaded and instantiated via their no-arg constructor.
The runner will invoke
execute on each instantiated
passing in the dispatch reporter to each
Runner is intended to be used from the command line. It is included in
package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take
over the role of runner, object
org.scalatest.tools.Runner may be excluded from that implementation of the package.
All other public types declared in package
org.scalatest.tools.Runner should be included in any such usage, however,
so client software can count on them being available.
Specifying "members-only" and "wildcard"
If you specify
Suite path names with
Runner will automatically
discover and execute accessible
Suites in the runpath that are either a member of (in the case of
or enclosed by (in the case of
-w) the specified path. As used in this context, a path is a portion of a fully qualified name.
For example, the fully qualifed name
com.example.webapp.MySuite contains paths
The fully qualifed name
com.example.webapp.MyObject.NestedSuite contains paths
Suite is a public class that extends
and defines a public no-arg constructor. Note that
Suites defined inside classes and traits do not have no-arg constructors,
and therefore won't be discovered.
Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus
they can be discovered.
For example, if you specify
on the command line, and you've placed
on the runpath, then
Runner will instantiate and execute both of those
Suites. The difference
-w is that for
Suites that are direct members of the named path
will be discovered. For
Suites whose fully qualified
name begins with the specified path will be discovered. Thus, if
exists on the runpath, invoking
-w com.example.webapp will cause
to be discovered, because its fully qualifed name begins with
"com.example.webapp". But if you invoke
GreenSuite will not be discovered because it is directly
a member of
If you specify no
-w arguments on the command line to
Runner, it will discover and execute all accessible
in the runpath.
Specifying TestNG XML config file paths
If you specify one or more file paths with
Runner will create a
passing in a
List of the specified paths. When executed, the
TestNGWrapperSuite will create one
and pass each specified file path to it for running. If you include
-t arguments, you must include TestNG's jar file on the class path or runpath.
-t argument will enable you to run existing
TestNG tests, including tests written in Java, as part of a ScalaTest run.
You need not use
-t to run suites written in Scala that extend
TestNGSuite. You can simply run such suites with
-m, or -w parameters.
Specifying JUnit tests
JUnit tests, including ones written in Java, may be run by specifying
-j classname, where the classname is a valid JUnit class
such as a TestCase, TestSuite, or a class implementing a static suite()
method returning a TestSuite.
To use this option you must include a JUnit jar file on your classpath.
(args : scala.Array[java.lang.String]) : Unit
Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details.
(args : scala.Array[java.lang.String]) : Boolean
Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details. The difference between this method and
|Methods inherited from AnyRef|
|getClass, hashCode, equals, clone, toString, notify, notifyAll, wait, wait, wait, finalize, ==, !=, eq, ne, synchronized|
|Methods inherited from Any|
|==, !=, isInstanceOf, asInstanceOf|
mainis simply that this method will block until the run has completed, aborted, or been stopped, and return
trueif all tests executed and passed. In other words, if any test fails, or if any suite aborts, or if the run aborts or is stopped, this method will return
false. This value is used, for example, by the ScalaTest ant task to determine whether to continue the build if
haltOnFailureis set to