org.scalatest.tools.Runner

Application that runs a suite of tests. The application accepts command line arguments that specify optional config map (key-value pairs), an optionalrunpath, zero to many Reporters, optional lists of tags to include and/or exclude, zero to manySuite class names, zero to many "members-only" Suite paths, zero to many "wildcard" Suite paths, and zero to many TestNG XML config file paths. All of these arguments are described in more detail below. Here's a summary:

scala [-classpath scalatest-&lt;version&gt;.jar:...] org.scalatest.tools.Runner
[-D&lt;key&gt;=&lt;value&gt; [...]] [-p &lt;runpath&gt;] [reporter [...]]
[-n &lt;includes&gt;] [-l &lt;excludes&gt;] [-c] [-s &lt;suite class name&gt;
[...]] [-j &lt;junit class name&gt; [...]] [-m &lt;members-only suite path&gt;
[...]] [-w &lt;wildcard suite path&gt; [...]] [-t &lt;TestNG config file
path&gt; [...]]

The simplest way to start Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:

scala -classpath scalatest-&lt;version&gt;.jar org.scalatest.tools.Runner -p compiled_tests

Given the previous command, Runner will discover and execute all Suites in the compiled_tests directory and its subdirectories, and show results in graphical user interface (GUI).

Specifying the config map

A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with "org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on theRunner command line. Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string. For example:

-Ddbname=testdb -Dserver=192.168.1.188

Specifying a runpath

A runpath is the list of filenames, directory paths, and/or URLs that Runneruses to load classes for the running test. If runpath is specified, Runner creates a custom class loader to load classes available on the runpath. The graphical user interface reloads the test classes anew for each run by creating and using a new instance of the custom class loader for each run. The classes that comprise the test may also be made available on the classpath, in which case no runpath need be specified.

The runpath is specified with the -p option. The -p must be followed by a space, a double quote ("), a white-space-separated list of paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off the double quotes, which only serve to combine a white-space separated list of strings into one command line argument. If you have path elements that themselves have a space in them, you must place a backslash (\) in front of the space. Here's an example:

-p "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"

Specifying reporters

Reporters can be specified on the command line in any of the following ways:

-g[configs...] - causes display of a graphical user interface that allows tests to be run and results to be investigated
-f[configs...] <filename> - causes test results to be written to the named file
-u <directory> - causes test results to be written to xml files in the named directory
-o[configs...] - causes test results to be written to the standard output
-e[configs...] - causes test results to be written to the standard error
-r[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

The [configs...] parameter, which is used to configure reporters, is described in the next section.

The -r option causes the reporter specified in<reporterclass> to be instantiated. Each reporter class specified with a -r option must be public, implementorg.scalatest.Reporter, and have a public no-arg constructor. Reporter classes must be specified with fully qualified names. The specified reporter classes may be deployed on the classpath. If a runpath is specified with the-p option, the specified reporter classes may also be loaded from the runpath. All specified reporter classes will be loaded and instantiated via their no-arg constructor.

For example, to run a suite named MySuite from the mydir directory using two reporters, the graphical reporter and a file reporter writing to a file named "test.out", you would type:

java -jar scalatest.jar -p mydir -g -f test.out -s MySuite

The -g, -o, or -e options can appear at most once each in any single command line. Multiple appearances of -f and -r result in multiple reporters unless the specified <filename> or <reporterclass> is repeated. If any of -g, -o, -e,<filename> or <reporterclass> are repeated on the command line, the Runner will print an error message and not run the tests.

Runner adds the reporters specified on the command line to a dispatch reporter, which will dispatch each method invocation to each contained reporter. Runner will pass the dispatch reporter to executed suites. As a result, every specified reporter will receive every report generated by the running suite of tests. If no reporters are specified, a graphical runner will be displayed that provides a graphical report of executed suites.

Configuring Reporters

Each reporter option on the command line can include configuration characters. Configuration characters are specified immediately following the -g, -o,-e, -f, or -r. The following configuration characters, which cause reports to be dropped, are valid for any reporter:

N - drop TestStarting events
C - drop TestSucceeded events
X - drop TestIgnored events
E - drop TestPending events
H - drop SuiteStarting events
L - drop SuiteCompleted events
O - drop InfoProvided events

A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not present information about the event in its report. For example, if you specify -oN, the standard output reporter will never receive any TestStarting events and will therefore never report them. The purpose of these configuration parameters is to allow users to selectively remove events they find add clutter to the report without providing essential information.

The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:

W - without color
D - show all durations
S - show short stack traces
F - show full stack traces

If you specify a W, D, S, or F for any reporter other than standard output, standard error, or file reporters, Runnerwill complain with an error message and not perform the run.

Configuring a standard output, error, or file reporter with D will cause that reporter to print a duration for each test and suite. When running in the default mode, a duration will only be printed for the entire run.

Configuring a standard output, error, or file reporter with F will cause that reporter to print full stack traces for all exceptions, including TestFailedExceptions. Every TestFailedException contains a stack depth of the line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default, mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown by production code. When a TestFailedException is thrown in default mode, only the source filename and line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack trace.

By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset terminal colors. Information printed as a result of run starting, completed, and stopped events is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green. The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode ('W') will turn off this behavior. No ansi codes will be inserted.

For example, to run a suite using two reporters, the graphical reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would type:

scala -classpath scalatest-<version>.jar -p mydir -g -eNDXEHLO -s MySuite

Note that no white space is allowed between the reporter option and the initial configuration parameters. So "-e NDXEHLO" will not work,"-eNDXEHLO" will work.

Specifying tags to include and exclude

You can specify tag names of tests to include or exclude from a run. To specify tags to include, use -n followed by a white-space-separated list of tag names to include, surrounded by double quotes. (The double quotes are not needed if specifying just one tag.) Similarly, to specify tags to exclude, use -l followed by a white-space-separated list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed if specifying just one tag.) If tags to include is not specified, then all tests except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed. (In other words, the absence of a -n option is like a wildcard, indicating all tests be included.) If tags to include is specified, then only those tests whose tags are mentioned in the argument following -nand not mentioned in the tags to exclude, will be executed. For more information on test tags, see the documentation for Suite. Here are some examples:

-n CheckinTests
-n FunctionalTests -l SlowTests
-n "CheckinTests FunctionalTests" -l "SlowTests NetworkTests"

Executing `Suite`s in parallel

With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run tests in parallel. If you include -c on the command line, Runner will pass a Distributor to the Suites you specify with -s. Runner will set up a thread pool to execute any Suites passed to the Distributor's put method in parallel. Trait Suite's implementation ofrunNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suiteof tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.

The -c option may optionally be appended with a number (e.g. "-c10" -- no intervening space) to specify the number of threads to be created in the thread pool. If no number (or 0) is specified, the number of threads will be decided based on the number of processors available.

Specifying `Suite`s

Suites are specified on the command line with a -s followed by the fully qualified name of a Suite subclass, as in:

-s com.artima.serviceuitest.ServiceUITestkit

Each specified suite class must be public, a subclass oforg.scalatest.Suite, and contain a public no-arg constructor.Suite classes must be specified with fully qualified names. The specified Suite classes may be loaded from the classpath. If a runpath is specified with the-p option, specified Suite classes may also be loaded from the runpath. All specified Suite classes will be loaded and instantiated via their no-arg constructor.

The runner will invoke execute on each instantiated org.scalatest.Suite, passing in the dispatch reporter to each execute method.

Runner is intended to be used from the command line. It is included in org.scalatestpackage as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take over the role of runner, object org.scalatest.tools.Runner may be excluded from that implementation of the package. All other public types declared in package org.scalatest.tools.Runner should be included in any such usage, however, so client software can count on them being available.

Specifying "members-only" and "wildcard" `Suite` paths

If you specify Suite path names with -m or -w, Runner will automatically discover and execute accessible Suites in the runpath that are either a member of (in the case of -m) or enclosed by (in the case of -w) the specified path. As used in this context, a path is a portion of a fully qualified name. For example, the fully qualifed name com.example.webapp.MySuite contains paths com, com.example, and com.example.webapp. The fully qualifed name com.example.webapp.MyObject.NestedSuite contains paths com, com.example,com.example.webapp, and com.example.webapp.MyObject. An accessible Suite is a public class that extends org.scalatest.Suiteand defines a public no-arg constructor. Note that Suites defined inside classes and traits do not have no-arg constructors, and therefore won't be discovered. Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus they can be discovered.

For example, if you specify -m com.example.webappon the command line, and you've placed com.example.webapp.RedSuite and com.example.webapp.BlueSuiteon the runpath, then Runner will instantiate and execute both of those Suites. The difference between -m and -w is that for -m, only Suites that are direct members of the named path will be discovered. For -w, any Suites whose fully qualified name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuiteexists on the runpath, invoking Runner with -w com.example.webapp will cause GreenSuiteto be discovered, because its fully qualifed name begins with "com.example.webapp". But if you invoke Runnerwith -m com.example.webapp, GreenSuite will not be discovered because it is directly a member of com.example.webapp.controllers, not com.example.webapp.

If you specify no -s, -m, or -w arguments on the command line to Runner, it will discover and execute all accessible Suites in the runpath.

Specifying TestNG XML config file paths

If you specify one or more file paths with -t, Runner will create a org.scalatest.testng.TestNGWrapperSuite, passing in a List of the specified paths. When executed, the TestNGWrapperSuite will create one TestNG instance and pass each specified file path to it for running. If you include -t arguments, you must include TestNG's jar file on the class path or runpath. The -t argument will enable you to run existing TestNG tests, including tests written in Java, as part of a ScalaTest run. You need not use -t to run suites written in Scala that extend TestNGSuite. You can simply run such suites with-s, -m, or -w parameters.

Specifying JUnit tests

JUnit tests, including ones written in Java, may be run by specifying-j classname, where the classname is a valid JUnit class such as a TestCase, TestSuite, or a class implementing a static suite() method returning a TestSuite.

To use this option you must include a JUnit jar file on your classpath.

linear super types: AnyRef, Any

Ordering

Alphabetic
By inheritance

Inherited

Hide All
Show all

Runner
AnyRef
Any

Visibility

Public
All

Impl.

Concrete
Abstract

Value Members

def != (arg0: AnyRef) : Boolean

attributes: final
definition classes: AnyRef
def != (arg0: Any) : Boolean

o != arg0 is the same as !(o == (arg0)).

o != arg0 is the same as !(o == (arg0)).
arg0
the object to compare against this object for dis-equality.
returns
false if the receiver object is equivalent to the argument; true otherwise.
attributes: final
definition classes: Any
def ## () : Int

attributes: final
definition classes: AnyRef → Any
def $asInstanceOf [T0] () : T0

attributes: final
definition classes: AnyRef
def $isInstanceOf [T0] () : Boolean

attributes: final
definition classes: AnyRef
def == (arg0: AnyRef) : Boolean

o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).
arg0
the object to compare against this object for equality.
returns
true if the receiver object is equivalent to the argument; false otherwise.
attributes: final
definition classes: AnyRef
def == (arg0: Any) : Boolean

o == arg0 is the same as o.equals(arg0).

o == arg0 is the same as o.equals(arg0).
arg0
the object to compare against this object for equality.
returns
true if the receiver object is equivalent to the argument; false otherwise.
attributes: final
definition classes: Any
def asInstanceOf [T0] : T0

This method is used to cast the receiver object to be of type T0.

This method is used to cast the receiver object to be of type T0.
Note that the success of a cast at runtime is modulo Scala's erasure semantics. Therefore the expression1.asInstanceOf[String] will throw a ClassCastException at runtime, while the expressionList(1).asInstanceOf[List[String]] will not. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.
returns
the receiver object.
attributes: final
definition classes: Any
def clone () : AnyRef

This method creates and returns a copy of the receiver object.

This method creates and returns a copy of the receiver object.
The default implementation of the clone method is platform dependent.
returns
a copy of the receiver object.
attributes: protected
definition classes: AnyRef
def eq (arg0: AnyRef) : Boolean

This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

This method is used to test whether the argument (arg0) is a reference to the receiver object (this).
The eq method implements an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation] on non-null instances of AnyRef: * It is reflexive: for any non-null instance x of type AnyRef, x.eq(x) returns true. * It is symmetric: for any non-null instances x and y of type AnyRef, x.eq(y) returns true if and only if y.eq(x) returns true. * It is transitive: for any non-null instances x, y, and z of type AnyRef if x.eq(y) returns true and y.eq(z) returns true, then x.eq(z) returns true.
Additionally, the eq method has three other properties. * It is consistent: for any non-null instances x and y of type AnyRef, multiple invocations of x.eq(y) consistently returns true or consistently returns false. * For any non-null instance x of type AnyRef, x.eq(null) and null.eq(x) returns false. * null.eq(null) returns true.
When overriding the equals or hashCode methods, it is important to ensure that their behavior is consistent with reference equality. Therefore, if two objects are references to each other (o1 eq o2), they should be equal to each other (o1 == o2) and they should hash to the same value (o1.hashCode == o2.hashCode).
arg0
the object to compare against this object for reference equality.
returns
true if the argument is a reference to the receiver object; false otherwise.
attributes: final
definition classes: AnyRef
def equals (arg0: Any) : Boolean

This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.
The default implementations of this method is an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation]: * It is reflexive: for any instance x of type Any, x.equals(x) should return true. * It is symmetric: for any instances x and y of type Any, x.equals(y) should return true if and only if y.equals(x) returns true. * It is transitive: for any instances x, y, and z of type AnyRef if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true.
If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is often necessary to override hashCode to ensure that objects that are "equal" (o1.equals(o2) returns true) hash to the same scala.Int (o1.hashCode.equals(o2.hashCode)).
arg0
the object to compare against this object for equality.
returns
true if the receiver object is equivalent to the argument; false otherwise.
definition classes: AnyRef → Any
def finalize () : Unit

This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.
The details of when and if the finalize method are invoked, as well as the interaction between finalizeand non-local returns and exceptions, are all platform dependent.
attributes: protected
definition classes: AnyRef
def getClass () : java.lang.Class[_]

Returns a representation that corresponds to the dynamic class of the receiver object.

Returns a representation that corresponds to the dynamic class of the receiver object.
The nature of the representation is platform dependent.
returns
a representation that corresponds to the dynamic class of the receiver object.
attributes: final
definition classes: AnyRef
def hashCode () : Int

Returns a hash code value for the object.

Returns a hash code value for the object.
The default hashing algorithm is platform dependent.
Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.
returns
the hash code value for the object.
definition classes: AnyRef → Any
def isInstanceOf [T0] : Boolean

This method is used to test whether the dynamic type of the receiver object is T0.

This method is used to test whether the dynamic type of the receiver object is T0.
Note that the test result of the test is modulo Scala's erasure semantics. Therefore the expression1.isInstanceOf[String] will return false, while the expression List(1).isInstanceOf[List[String]] will return true. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.
returns
true if the receiver object is an instance of erasure of type T0; false otherwise.
attributes: final
definition classes: Any
def main (args: Array[String]) : Unit

Runs a suite of tests, with optional GUI.

Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details.
def ne (arg0: AnyRef) : Boolean

o.ne(arg0) is the same as !(o.eq(arg0)).

o.ne(arg0) is the same as !(o.eq(arg0)).
arg0
the object to compare against this object for reference dis-equality.
returns
false if the argument is not a reference to the receiver object; true otherwise.
attributes: final
definition classes: AnyRef
def notify () : Unit

Wakes up a single thread that is waiting on the receiver object's monitor.

Wakes up a single thread that is waiting on the receiver object's monitor.
attributes: final
definition classes: AnyRef
def notifyAll () : Unit

Wakes up all threads that are waiting on the receiver object's monitor.

Wakes up all threads that are waiting on the receiver object's monitor.
attributes: final
definition classes: AnyRef
def run (args: Array[String]) : Boolean

Runs a suite of tests, with optional GUI.

Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details. The difference between this method and main is simply that this method will block until the run has completed, aborted, or been stopped, and return true if all tests executed and passed. In other words, if any test fails, or if any suite aborts, or if the run aborts or is stopped, this method will return false. This value is used, for example, by the ScalaTest ant task to determine whether to continue the build if haltOnFailure is set to true.
returns
true if all tests were executed and passed.
def synchronized [T0] (arg0: T0) : T0

attributes: final
definition classes: AnyRef
def toString () : String

Returns a string representation of the object.

Returns a string representation of the object.
The default representation is platform dependent.
returns
a string representation of the object.
definition classes: AnyRef → Any
def wait () : Unit

attributes: final
definition classes: AnyRef
def wait (arg0: Long, arg1: Int) : Unit

attributes: final
definition classes: AnyRef
def wait (arg0: Long) : Unit

attributes: final
definition classes: AnyRef

Runner

object Runner extends AnyRef

Specifying the config map

Specifying a runpath

Specifying reporters

Configuring Reporters

Specifying tags to include and exclude

Executing `Suite`s in parallel

Specifying `Suite`s

Specifying "members-only" and "wildcard" `Suite` paths

Specifying TestNG XML config file paths

Specifying JUnit tests

Value Members

def != (arg0: AnyRef) : Boolean

def != (arg0: Any) : Boolean

def ## () : Int

def $asInstanceOf [T0] () : T0

def $isInstanceOf [T0] () : Boolean

def == (arg0: AnyRef) : Boolean

def == (arg0: Any) : Boolean

def asInstanceOf [T0] : T0

def clone () : AnyRef

def eq (arg0: AnyRef) : Boolean

def equals (arg0: Any) : Boolean

def finalize () : Unit

def getClass () : java.lang.Class[_]

def hashCode () : Int

def isInstanceOf [T0] : Boolean

def main (args: Array[String]) : Unit

def ne (arg0: AnyRef) : Boolean

def notify () : Unit

def notifyAll () : Unit

def run (args: Array[String]) : Boolean

def synchronized [T0] (arg0: T0) : T0

def toString () : String

def wait () : Unit

def wait (arg0: Long, arg1: Int) : Unit

def wait (arg0: Long) : Unit

Inherited from AnyRef

Inherited from Any

Runner

object Runner extends AnyRef

Specifying the config map

Specifying a runpath

Specifying reporters

Configuring Reporters

Specifying tags to include and exclude

Executing Suites in parallel

Specifying Suites

Specifying "members-only" and "wildcard" Suite paths

Specifying TestNG XML config file paths

Specifying JUnit tests

Value Members

def != (arg0: AnyRef) : Boolean

def != (arg0: Any) : Boolean

def ## () : Int

def $asInstanceOf [T0] () : T0

def $isInstanceOf [T0] () : Boolean

def == (arg0: AnyRef) : Boolean

def == (arg0: Any) : Boolean

def asInstanceOf [T0] : T0

def clone () : AnyRef

def eq (arg0: AnyRef) : Boolean

def equals (arg0: Any) : Boolean

def finalize () : Unit

def getClass () : java.lang.Class[_]

def hashCode () : Int

def isInstanceOf [T0] : Boolean

def main (args: Array[String]) : Unit

def ne (arg0: AnyRef) : Boolean

def notify () : Unit

def notifyAll () : Unit

def run (args: Array[String]) : Boolean

def synchronized [T0] (arg0: T0) : T0

def toString () : String

def wait () : Unit

def wait (arg0: Long, arg1: Int) : Unit

def wait (arg0: Long) : Unit

Inherited from AnyRef

Inherited from Any

Executing `Suite`s in parallel

Specifying `Suite`s

Specifying "members-only" and "wildcard" `Suite` paths