This class is ScalaTest's implementation of the new Framework API that is supported in sbt 0.13.
An ant task to run ScalaTest.
An ant task to run ScalaTest. Instructions on how to specify various
options are below. See the main documentation for object Runner
class for a description
of what each of the options does.
To use the ScalaTest ant task, you must first define it in your ant file using taskdef
.
Here's an example:
<path id="scalatest.classpath"> <pathelement location="${lib}/scalatest.jar"/> <pathelement location="${lib}/scala-library.jar"/> <-- scala-actors.jar needed only for ScalaTest <= 1.9.1 on Scala >= 2.10.0 --> <pathelement location="${lib}/scala-actors.jar"/> </path> <target name="main" depends="dist"> <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestAntTask"> <classpath refid="scalatest.classpath"/> </taskdef> <scalatest ... </target>
Note that you only need the scala-actors.jar
if you are using ScalaTest version 1.9.1 or earlier
with Scala 2.10 or later.
Once defined, you use the task by specifying information in a scalatest
element:
<scalatest ...> ... </scalatest>
You can place key value pairs into the config map using nested <config>
elements,
like this:
<scalatest> <config name="dbname" value="testdb"/> <config name="server" value="192.168.1.188"/>
You can specify a runpath using either a runpath
attribute and/or nested
<runpath>
elements, using standard ant path notation:
<scalatest runpath="serviceuitest-1.1beta4.jar:myjini">
or
<scalatest> <runpath> <pathelement location="serviceuitest-1.1beta4.jar"/> <pathelement location="myjini"/> </runpath>
To add a URL to your runpath, use a <runpathurl>
element
(since ant paths don't support URLs):
<scalatest> <runpathurl url="http://foo.com/bar.jar"/>
You can specify reporters using nested <reporter>
elements, where the type
attribute must be one of the following:
graphic
file
memory
junitxml
html
stdout
stderr
reporterclass
Each may include a config
attribute to specify the reporter configuration.
Types file
, memory
, junitxml
, html
, and reporterclass
require additional attributes (the css attribute is optional for the html reporter):
<scalatest> <reporter type="stdout" config="FD"/> <reporter type="file" filename="test.out"/> <reporter type="memory" filename="target/memory.out"/> <reporter type="junitxml" directory="target"/> <reporter type="html" directory="target" css="src/main/html/mystylesheet.css"/> <reporter type="reporterclass" classname="my.ReporterClass"/>
Specify tags to include and/or exclude using <tagsToInclude>
and
<tagsToExclude>
elements, like this:
<scalatest> <tagsToInclude> CheckinTests FunctionalTests </tagsToInclude> <tagsToExclude> SlowTests NetworkTests </tagsToExclude>
Tags to include or exclude can also be specified using attributes tagsToInclude and tagsToExclude, with arguments specified as whitespace- delimited lists.
To specify suites to run, use either a suite
attribute or nested
<suite>
elements:
<scalatest suite="com.artima.serviceuitest.ServiceUITestkit">
or
<scalatest> <suite classname="com.artima.serviceuitest.ServiceUITestkit"/>
To specify tests to run, use nested <test>
elements with
either a 'name' or 'substring' attribute:
<scalatest> <test name="hello test"/> <test substring="hello"/>
To specify suites using members-only or wildcard package names, use
either the membersonly
or wildcard
attributes, or nested
<membersonly>
or <wildcard>
elements:
<scalatest membersonly="com.artima.serviceuitest">
or
<scalatest wildcard="com.artima.joker">
or
<scalatest> <membersonly package="com.artima.serviceuitest"/> <wildcard package="com.artima.joker"/>
Use attribute suffixes="[pipe-delimited list of suffixes]"
to specify that only classes whose names end in one of the specified suffixes
should be included in discovery searches for Suites to test. This can
be used to improve discovery time or to limit the scope of a test. E.g.:
<scalatest suffixes="Spec|Suite">
Use attribute testsfile="[file name]"
or nested
<testsfile> elements to specify files containing a list of
tests to be run. This is used to rerun failed/canceled tests
listed in files written by the memory reporter. E.g.:
<scalatest testsfile="target/memory.out">
or
<scalatest> <testsfile filename="target/memory.out"/>
Use attribute parallel="true"
to specify parallel execution of suites.
(If the parallel
attribute is left out or set to false, suites will be executed sequentially by one thread.)
When parallel
is true, you can include an optional sortSuites
attribute to request that events be sorted on-the-fly so that
events for the same suite are reported together, with a timeout, (e.g., sortSuites="true"
),
and an optional numthreads
attribute to specify the number
of threads to be created in thread pool (e.g., numthreads="10"
).
Use attribute haltonfailure="true"
to cause ant to fail the
build if there's a test failure.
Use attribute fork="true"
to cause ant to run the tests in
a separate process.
When fork
is true
, attribute maxmemory
may be used to specify
the maximum memory size that will be passed to the forked jvm. For example, the following setting
will cause "-Xmx1280M"
to be passed to the java command used to
run the tests.
<scalatest maxmemory="1280M">
When fork
is true, nested <jvmarg>
elements may be used
to pass additional arguments to the forked jvm.
For example, if you are running into 'PermGen space' memory errors,
you could add the following jvmarg
to bump up the JVM's MaxPermSize
value:
<jvmarg value="-XX:MaxPermSize=128m"/>
Class that makes ScalaTest tests visible to SBT (prior to version 0.13).
Class that makes ScalaTest tests visible to SBT (prior to version 0.13).
To use ScalaTest in SBT, you should add ScalaTest as dependency in your SBT build file, the following shows an example for using ScalaTest 2.0 with Scala 2.10.x project:
"org.scalatest" % "scalatest_2.10" % "2.0" % "test"
To pass argument to ScalaTest from SBT, you can use testOptions
:
testOptions in Test += Tests.Argument("-u", "target/junit") // Use JUnitXmlReporter
If you are using multiple testing frameworks, you can pass arguments specific to ScalaTest only:
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/junit") // Use JUnitXmlReporter
Integration in SBT 0.13 supports same argument format as Runner
,
except the following arguments:
-R
-- runpath is not supported because test path and discovery is handled by SBT-s
-- suite is not supported because SBT's test-only
serves the similar purpose-A
-- again is not supported because SBT's test-quick
serves the similar purpose-j
-- junit is not supported because in SBT different test framework should be supported by its corresponding Framework
implementation-b
-- testng is not supported because in SBT different test framework should be supported by its corresponding Framework
implementation-P
-- concurrent/parallel is not supported because parallel execution is controlled by SBT.-q
is not supported because test discovery should be handled by SBT, and SBT's test-only or test filter serves the similar purpose-T
is not supported because correct ordering of text output is handled by SBT-g
is not supported because current Graphic Reporter implementation works differently than standard reporter
It is highly recommended to upgrade to SBT 0.13 to enjoy the best of ScalaTest 2.0 SBT integration. Due to limitations
in old Framework API (prior to SBT 0.13), it is hard to support ScalaTest features in the most efficient way. One example is
the nested suites, where in old Framework API they has to be executed sequentially, while new Framework API (included in SBT
0.13) the concept of nested Task
has enabled parallel execution of ScalaTest's nested suites.
Application that runs a suite of tests.
Application that runs a suite of tests.
Note: this application offers the full range of ScalaTest features via command line arguments described below. If you just want to run a suite of tests from the command line and see results on the standard output, you may prefer to use ScalaTest's simple runner.
The basic form of a Runner
invocation is:
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
The arguments Runner
accepts are described in the following table:
argument | description | example |
---|---|---|
-Dkey=value | defines a key/value pair for the config map | -DmaxConnections=100 |
-R <runpath elements> | the specifies the runpath from which tests classes will be discovered and loaded (Note: only one -R allowed) | Unix: -R target/classes:target/generated/classes Windows: -R target\classes;target\generated\classes |
-n <tag name> | specifies a tag to include (Note: only one tag name allowed per -n ) | -n UnitTests -n FastTests |
-l <tag name> | specifies a tag to exclude (Note: only one tag name allowed per -l ) | -l SlowTests -l PerfTests |
-P[S][integer thread count] | specifies a parallel run, with optional suite sorting and thread count (Note: only one -P allowed) | -P , -PS , -PS 8 , or -P8 |
-s <suite class name> | specifies a suite class to run | -s com.company.project.StackSpec |
-m <members-only package> | requests that suites that are direct members of the specified package be discovered and run | -m com.company.project |
-w <wildcard package> | requests that suites that are members of the specified package or its subpackages be discovered and run | -w com.company.project |
-q <suffixes> | specify suffixes to discover | -q Spec -q Suite |
-Q | discover only classes whose names end with Spec or Suite (or other suffixes specified by -q ) | -Q |
-j <JUnit class name> | instantiate and run a JUnit test class | -j StackTestClass |
-b <TestNG XML file> | run TestNG tests using the specified TestNG XML file | -b testng.xml |
-F <span scale factor> | a factor by which to scale time spans (Note: only one -F is allowed) | -F 10 or -F 2.5 |
-T <sorting timeout> | specifies a integer timeout (in seconds) for sorting the events of parallel runs back into sequential order | -T 5 |
-y <chosen styles> | specifies chosen styles for your project | -y org.scalatest.FlatSpec |
-i <suite ID> | specifies a suite to run by ID (Note: must follow -s , and is intended to be used primarily by tools such as IDEs.) | -i com.company.project.FileSpec-file1.txt |
-t <test name> | select the test with the specified name | -t "An empty Stack should complain when popped" |
-z <test name substring> | select tests whose names include the specified substring | -z "popped" |
-g[NCXEHLOPQMD] | select the graphical reporter | -g |
-f[NCXEHLOPQMDWSFU] <filename> | select the file reporter | -f output.txt |
-u <directory name> | select the JUnit XML reporter | -u target/junitxmldir |
-h <directory name> [-Y <css file name>] | select the HTML reporter, optionally including the specified CSS file | -h target/htmldir -Y src/main/html/customStyles.css |
-v | print the ScalaTest version | -v or, also -version |
-o[NCXEHLOPQMDWSFU] | select the standard output reporter | -o |
-e[NCXEHLOPQMDWSFU] | select the standard error reporter | -e |
-C[NCXEHLOPQMD] <reporter class> | select a custom reporter | -C com.company.project.BarReporter |
-M <file name> | memorize failed and canceled tests in a file, so they can be rerun with -A (again) | -M rerun.txt |
-A <file name> | used in conjunction with -M (momento) to select previously failed and canceled tests to rerun again | -A rerun.txt |
-W <delay> <period> | requests notifications of slowpoke tests, tests that have been running longer than delay seconds, every period seconds. | -W 60 60 |
The simplest way to start Runner
is to specify the directory containing your compiled tests as the sole element of the runpath, for example:
scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -R compiled_tests
Given the previous command, Runner
will discover and execute all Suite
s in the compiled_tests
directory and its subdirectories,
and show results in graphical user interface (GUI).
Each -s
argument must be followed by one and only one fully qualified class name. The class must either extend Suite
and
have a public, no-arg constructor, or be annotated by a valid WrapWith
annotation.
A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with
"org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on the
Runner
command line.
Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string.
For example:
-Ddbname=testdb -Dserver=192.168.1.188
A runpath is the list of filenames, directory paths, and/or URLs that Runner
uses to load classes for the running test. If runpath is specified, Runner
creates
a custom class loader to load classes available on the runpath.
The graphical user interface reloads the test classes anew for each run
by creating and using a new instance of the custom class loader for each run.
The classes that comprise the test may also be made available on
the classpath, in which case no runpath need be specified.
The runpath is specified with the -R option. The -R must be followed by a space,
a double quote ("
), a white-space-separated list of
paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off
the double quotes, which only serve to combine a white-space separated list of strings into one
command line argument. If you have path elements that themselves have a space in them, you must
place a backslash (\) in front of the space. Here's an example:
-R "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"
Reporters can be specified on the command line in any of the following ways:
-g[configs...]
- causes display of a graphical user interface that allows
tests to be run and results to be investigated-f[configs...] <filename>
- causes test results to be written to
the named file-u <directory>
- causes test results to be written to
junit-style xml files in the named directory-h <directory> [-Y <CSS file>]
- causes test results to be written to
HTML files in the named directory, optionally included the specified CSS file-a <number of files to archive>
- causes specified number of old
summary and durations files to be archived (in summaries/ and durations/ subdirectories)
for dashboard reporter (default is two)-o[configs...]
- causes test results to be written to
the standard output-e[configs...]
- causes test results to be written to
the standard error-k <host> <port>
- causes test results to be written to
socket in the named host and port number, using XML format-K <host> <port>
- causes test results to be written to
socket in the named host and port number, using Java object binary format-C[configs...] <reporterclass>
- causes test results to be reported to
an instance of the specified fully qualified Reporter
class nameThe [configs...]
parameter, which is used to configure reporters, is described in the next section.
The -C
option causes the reporter specified in
<reporterclass>
to be
instantiated.
Each reporter class specified with a -C option must be public, implement
org.scalatest.Reporter
, and have a public no-arg constructor.
Reporter classes must be specified with fully qualified names.
The specified reporter classes may be
deployed on the classpath. If a runpath is specified with the
-R
option, the specified reporter classes may also be loaded from the runpath.
All specified reporter classes will be loaded and instantiated via their no-arg constructor.
For example, to run a suite named MySuite
from the mydir
directory
using two reporters, the graphical reporter and a file reporter
writing to a file named "test.out"
, you would type:
java -jar scalatest.jar -R mydir -g -f test.out -s MySuite
The -g
, -o
, or -e
options can
appear at most once each in any single command line.
Multiple appearances of -f
and -C
result in multiple reporters
unless the specified <filename>
or <reporterclass>
is
repeated. If any of -g
, -o
, -e
,
<filename>
or <reporterclass>
are repeated on
the command line, the Runner
will print an error message and not run the tests.
Runner
adds the reporters specified on the command line to a dispatch reporter,
which will dispatch each method invocation to each contained reporter. Runner
will pass
the dispatch reporter to executed suites. As a result, every
specified reporter will receive every report generated by the running suite of tests.
If no reporters are specified, a graphical
runner will be displayed that provides a graphical report of
executed suites.
Each reporter option on the command line can include configuration characters. Configuration characters
are specified immediately following the -g
, -o
,
-e
, -f
, or -C
. The following configuration
characters, which cause reports to be dropped, are valid for any reporter:
N
- drop TestStarting
eventsC
- drop TestSucceeded
eventsX
- drop TestIgnored
eventsE
- drop TestPending
eventsH
- drop SuiteStarting
eventsL
- drop SuiteCompleted
eventsO
- drop InfoProvided
eventsP
- drop ScopeOpened
eventsQ
- drop ScopeClosed
eventsR
- drop ScopePending
eventsM
- drop MarkupProvided
eventsA dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not
present information about the event in its report. For example, if you specify -oN
, the standard output reporter
will never receive any TestStarting
events and will therefore never report them. The purpose of these
configuration parameters is to allow users to selectively remove events they find add clutter to the report without
providing essential information.
The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:
W
- without colorD
- show all durationsS
- show short stack tracesF
- show full stack tracesU
- unformatted modeI
- show reminder of failed and canceled tests without stack tracesT
- show reminder of failed and canceled tests with short stack tracesG
- show reminder of failed and canceled tests with full stack tracesK
- exclude TestCanceled
events from reminderIf you specify a W, D, S, F, U, R, T, G, or K for any reporter other than standard output, standard error, or file reporters, Runner
will complain with an error message and not perform the run.
Configuring a standard output, error, or file reporter with D
will cause that reporter to
print a duration for each test and suite. When running in the default mode, a duration will only be printed for
the entire run.
Configuring a standard output, error, or file reporter with F
will cause that reporter to print full stack traces for all exceptions,
including TestFailedExceptions
. Every TestFailedException
contains a stack depth of the
line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default,
mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown
by production code. When a TestFailedException
is thrown in default mode, only the source filename and
line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack
trace.
The 'U' unformatted configuration removes some formatting from the output and adds verbosity. The purpose of unformatted (or, "ugly") mode is to facilitate debugging of parallel runs. If you have tests that fail or hang during parallel runs, but succeed when run sequentially, unformatted mode can help. In unformatted mode, you can see exactly what is happening when it is happening. Rather than attempting to make the output look as pretty and human-readable as possible, unformatted mode will just print out verbose information about each event as it arrives, helping you track down the problem you are trying to debug.
By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset
terminal colors. Information printed as a result of run starting, completed, and stopped events
is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed
as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green.
The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can
get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode (W
) will
turn off this behavior. No ansi codes will be inserted.
The R
, T
, and G
options enable "reminders" of failed and, optionally, canceled tests to be printed
at the end of the summary. This minimizes or eliminates the need to search and scroll backwards to find out what tests failed or were canceled.
For large test suites, the actual failure message could have scrolled off the top of the buffer, making it otherwise impossible
to see what failed. You can configure the detail level of the stack trace for regular reports of failed and canceled tests independently
from that of reminders. To set the detail level for regular reports, use S
for short stack traces, F
for
full stack traces, or nothing for the default of no stack trace. To set the detail level for reminder reports, use T
for
reminders with short stack traces, G
for reminders with full stack traces in reminders, or R
for reminders
with no stack traces. If you wish to exclude reminders of canceled tests, i.e., only see reminders of failed tests, specify
K
along with one of R
, T
, or G
, as in "-oRK"
.
For example, to run a suite using two reporters, the graphical reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would type:
scala -classpath scalatest-<version>.jar -R mydir -g -eNDXEHLO -s MySuite
Note that no white space is allowed between the reporter option and the initial configuration
parameters. So "-e NDXEHLO"
will not work,
"-eNDXEHLO"
will work.
You can specify tag names of tests to include or exclude from a run. To specify tags to include,
use -n
followed by a white-space-separated list of tag names to include, surrounded by
double quotes. (The double quotes are not needed if specifying just one tag.) Similarly, to specify tags
to exclude, use -l
followed by a white-space-separated
list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed
if specifying just one tag.) If tags to include is not specified, then all tests
except those mentioned in the tags to exclude (and in the org.scalatest.Ignore
tag), will be executed.
(In other words, the absence of a -n
option is like a wildcard, indicating all tests be included.)
If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n
and not mentioned in the tags to exclude, will be executed. For more information on test tags, see
the documentation for Suite
. Here are some examples:
-n CheckinTests
-n FunctionalTests -l org.scalatest.tags.Slow
-n "CheckinTests FunctionalTests" -l "org.scalatest.tags.Slow org.scalatest.tags.Network"
You can specify suffixes of Suite
names to discover. To specify suffixes to discover,
use -q
followed by a vertical-bar-separated list of suffixes to discover, surrounded by
double quotes. (The double quotes are not needed if specifying just one suffix.) Or you can specify
them individually using multiple -q's.
If suffixes to discover is not specified, then all suffixes are considered.
If suffixes is specified, then only those Suites whose class names end in one of the specified suffixes
will be considered during discovery. Here are some examples:
-q Spec
-q "Spec|Suite"
-q Spec -q Suite
Option -Q can be used to specify a default set of suffixes "Spec|Suite". If you specify both -Q and -q, you'll get Spec and Suite in addition to the other suffix or suffixes you specify with -q.
Specifying suffixes can speed up the discovery process because class files with names not ending the specified suffixes
can be immediately disqualified, without needing to load and inspect them to see if they either extend Suite
and declare a public, no-arg constructor, or are annotated with WrapWith
.
Suite
s in parallel With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run
tests in parallel. If you include -P
on the command line, Runner
will pass a Distributor
to
the Suite
s you specify with -s
. Runner
will set up a thread pool to execute any Suite
s
passed to the Distributor
's put
method in parallel. Trait Suite
's implementation of
runNestedSuites
will place any nested Suite
s into this Distributor
. Thus, if you have a Suite
of tests that must be executed sequentially, you should override runNestedSuites
as described in the documentation for Distributor
.
The -P
option may optionally be appended with a number (e.g.
"-P10
" -- no intervening space) to specify the number of
threads to be created in the thread pool. If no number (or 0) is
specified, the number of threads will be decided based on the number of
processors available.
Suite
s Suites are specified on the command line with a -s followed by the fully qualified
name of a Suite
subclass, as in:
-s com.artima.serviceuitest.ServiceUITestkit
Each specified suite class must be public, a subclass of
org.scalatest.Suite
, and contain a public no-arg constructor.
Suite
classes must be specified with fully qualified names.
The specified Suite
classes may be
loaded from the classpath. If a runpath is specified with the
-R
option, specified Suite
classes may also be loaded from the runpath.
All specified Suite
classes will be loaded and instantiated via their no-arg constructor.
The runner will invoke execute
on each instantiated org.scalatest.Suite
,
passing in the dispatch reporter to each execute
method.
Runner
is intended to be used from the command line. It is included in org.scalatest
package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take
over the role of runner, object org.scalatest.tools.Runner
may be excluded from that implementation of the package.
All other public types declared in package org.scalatest.tools.Runner
should be included in any such usage, however,
so client software can count on them being available.
Suite
paths If you specify Suite
path names with -m
or -w
, Runner
will automatically
discover and execute accessible Suite
s in the runpath that are either a member of (in the case of -m
)
or enclosed by (in the case of -w
) the specified path. As used in this context, a path is a portion of a fully qualified name.
For example, the fully qualifed name com.example.webapp.MySuite
contains paths com
, com.example
, and com.example.webapp
.
The fully qualifed name com.example.webapp.MyObject.NestedSuite
contains paths com
, com.example
,
com.example.webapp
, and com.example.webapp.MyObject
.
An accessible Suite
is a public class that extends org.scalatest.Suite
and defines a public no-arg constructor. Note that Suite
s defined inside classes and traits do not have no-arg constructors,
and therefore won't be discovered. Suite
s defined inside singleton objects, however, do get a no-arg constructor by default, thus
they can be discovered.
For example, if you specify -m com.example.webapp
on the command line, and you've placed com.example.webapp.RedSuite
and com.example.webapp.BlueSuite
on the runpath, then Runner
will instantiate and execute both of those Suite
s. The difference
between -m
and -w
is that for -m
, only Suite
s that are direct members of the named path
will be discovered. For -w
, any Suite
s whose fully qualified
name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuite
exists on the runpath, invoking Runner
with -w com.example.webapp
will cause GreenSuite
to be discovered, because its fully qualifed name begins with "com.example.webapp"
. But if you invoke Runner
with -m com.example.webapp
, GreenSuite
will not be discovered because it is directly
a member of com.example.webapp.controllers
, not com.example.webapp
.
If you specify no -s
, -m
, or -w
arguments on the command line to Runner
, it will discover and execute all accessible Suite
s
in the runpath.
You can optionally specify chosen styles for a ScalaTest run. ScalaTest supports different styles of testing so that different teams can use the style or styles that best suits their situation and culture. But in any one project, it is recommended you decide on one main style for unit testing, and consistently use only that style for unit testing throughout the project. If you also have integration tests in your project, you may wish to pick a different style for them than you are using for unit testing. You may want to allow certain styles to be used in special testing situations on a project, but in general, it is best to minimize the styles used in any given project to a few, or one.
To facilitate the communication and enforcement of a team's style choices for a project, you can
specify the chosen styles in your project build. If chosen styles is defined, ScalaTest style traits that are
not among the chosen list will abort with a message complaining that the style trait is not one of the
chosen styles. The style name for each ScalaTest style trait is its fully qualified name. For example,
to specify that org.scalatest.FunSpec
as your chosen style you'd pass this to
Runner
:
-y org.scalatest.FunSpec
If you wanted org.scalatest.FunSpec
as your main unit testing style, but also wanted to
allow PropSpec
for test matrixes and FeatureSpec
for
integration tests, you would write:
-y org.scalatest.FunSpec -y org.scalatest.PropSpec -y org.scalatest.FeatureSpec
To select org.scalatest.FlatSpec
as your main unit testing style, but allow
org.scalatest.fixture.FlatSpec
for multi-threaded unit tests, you'd write:
-y org.scalatest.FlatSpec -y org.scalatest.fixture.FlatSpec
The style name for a suite is obtained by invoking its styleName
method. Custom style
traits can override this method so that a custom style can participate in the chosen styles list.
Because ScalaTest is so customizable, a determined programmer could circumvent
the chosen styles check, but in practice -y
should be persuasive enough tool
to keep most team members in line.
Runner
accepts three arguments that facilitate selecting suites and tests: -i
, -t
, and -z.
The -i
option enables a suite to be selected by suite ID. This argument is intended to allow tools such as IDEs or build tools to
rerun specific tests or suites from information included in the results of a previous run. A -i
must follow a -s
that specifies a class with a public, no-arg constructor. The -i
parameter can be used, for example, to rerun a nested suite that
declares no zero-arg constructor, which was created by containing suite that does declare a no-arg constructor. In this case, -s
would be
used to specify the class ScalaTest can instantiate directly, the containing suite that has a public, no-arg constructor, and -i
would be
used to select the desired nested suite. One important use case for -i
is to enable such a nested suite that aborted during the previous run
to be rerun.
The -t
argument allows a test to be selected by its (complete) test name. Like -i
, the -t
argument is primarily intented
to be used by tools such as IDEs or build tools, to rerun selected tests based on information obtained from the results of a previous run.
For example, -t
could be used to rerun a test that failed in the previous run.
The -t
argument can be used directly by users, but because descriptive test names are usually rather long, the -z
argument (described next), will
usually be a more practical choice for users. If a -t
follows either -s
or -i
, then it only applies to the suite
identified. If it is specified independent of a -s
or -i
, then discovery is performed to find all Suites containing the test name.
The -z
option allows tests to be selected by a simplified wildcard: any test whose name includes the substring specified after -z
will be selected. For example, -z popped
would select tests named "An empty stack should complain when popped"
and "A non-empty stack
should return the last-pushed value when popped
, but not "An empty stack should be empty"
. In short, -z popped
would select any
tests whose name includes the substring "popped"
, and not select any tests whose names don't include "popped"
. This simplified
approach to test name wildcards, which was suggested by Mathias Doenitz, works around the difficulty of finding an actual wildcard character that will work
reliably on different operating systems. Like -t
, if -z
follows -s
or -i
, then it only applies to the Suite specified. Otherwise discovery is performed to find all Suites containing test names that include the substring.
If you specify a integer or floating point span scale factor with -F
, trait ScaledTimeSpans
trait will return the specified value from its implementation of spanScaleFactor
. This allows you to tune the "patience" of a run (how long to wait
for asynchronous operations) from the command line. For more information, see the documentation for trait ScaledTimeSpans
.
If you specify one or more file paths with -b
(b for Beust, the last name of TestNG's creator), Runner
will create a org.scalatest.testng.TestNGWrapperSuite
,
passing in a List
of the specified paths. When executed, the TestNGWrapperSuite
will create one TestNG
instance
and pass each specified file path to it for running. If you include -b
arguments, you must include TestNG's jar file on the class path or runpath.
The -b
argument will enable you to run existing TestNG
tests, including tests written in Java, as part of a ScalaTest run.
You need not use -b
to run suites written in Scala that extend TestNGSuite
. You can simply run such suites with
-s
, -m
, or -w parameters.
JUnit tests, including ones written in Java, may be run by specifying
-j classname
, where the classname is a valid JUnit class
such as a TestCase, TestSuite, or a class implementing a static suite()
method returning a TestSuite.
To use this option you must include a JUnit jar file on your classpath.
You can memorize failed and canceled tests using -M
:
-M failed-canceled.txt
All failed and canceled tests will be memorized in failed-canceled.txt
, to rerun them again, you use -A
:
-A failed-canceled.txt
You can request to recieve periodic notifications of slowpokes, tests that have been running longer than a given amount of time, specified in
seconds by the first integer after -W
, the delay.
You specify the period between slowpoke notifications in seconds with the second integer after -W
, the period. Thus to receive
notifications very minute of tests that have been running longer than two minutes, you'd use:
-W 120 60
Slowpoke notifications will be sent via AlertProvided
events. The standard out reporter, for example,
will report such notifications like:
*** Test still running after 2 minutes, 13 seconds: suite name: ExampleSpec, test name: An egg timer should take 10 minutes.
This class is ScalaTest's implementation of the new Framework API that is supported in sbt 0.13.
To use ScalaTest in sbt, you should add ScalaTest as dependency in your sbt build file, the following shows an example for using ScalaTest 2.0 with Scala 2.10.x project:
To pass argument to ScalaTest from sbt, you can use
testOptions
:If you are using multiple testing frameworks, you can pass arguments specific to ScalaTest only:
Supported arguments
Integration in sbt 0.13 supports same argument format as
Runner
, except the following arguments:-R
-- runpath is not supported because test path and discovery is handled by sbt-s
-- suite is not supported because sbt'stest-only
serves the similar purpose-A
-- again is not supported because sbt'stest-quick
serves the similar purpose-j
-- junit is not supported because in sbt different test framework should be supported by its correspondingFramework
implementation-b
-- testng is not supported because in sbt different test framework should be supported by its correspondingFramework
implementation-P
-- concurrent/parallel is not supported because parallel execution is controlled by sbt.-q
is not supported because test discovery should be handled by sbt, and sbt's test-only or test filter serves the similar purpose-T
is not supported because correct ordering of text output is handled by sbt-g
is not supported because current Graphic Reporter implementation works differently than standard reporterNew Features of New Framework API
New Framework API supports a number of new features that ScalaTest has utilized to support a better testing experience in sbt. The followings are summary of new features supported by the new Framework API:
Runner
per project run (non-fork), and a newdone
methodfork
modeIgnored
,Canceled
andPending
statusSpecified behavior of single instance of
Runner
per project run (non-fork), and a newdone
methodIn new Framework API, it is now a specified behavior that
Framework
'srunner
method will be called to get aRunner
instance once per project run. Arguments will be passed when callingFramework
'srunner
and this gives ScalaTest a good place to perform setup tasks, such as initializingReporter
s.There's also a new
done
onRunner
interface, which in turns provide a good spot for ScalaTest to perform cleanup tasks, such as disposing theReporter
s.HtmlReporter
depends on this behavior to generate itsindex.html
. In addition,done
can return framework-specific summary text for sbt to render at the end of the project run, which allows ScalaTest to return its own summary text.API to return nested
Suite
s as sbtTask
sIn new Framework API, a new concept of
Task
was introduced. ATask
has anexecute
method that can return moreTask
s for execution. ScalaTest does not utilize this feature, it always return empty array for sub-tasks.API to support test execution in
fork
modeForking was added to sbt since version 0.12, you can find documentation for forking support in sbt at Forking in sbt.
Although forking is already available in sbt since 0.12, there's no support in old Framework API, until it is added in new Framework API that is supported in sbt 0.13. With API provided with new Framework API, ScalaTest creates real
Reporter
s in the main process, and usesSocketReporter
in forked process to send events back to the main process, and get processed by realReporter
s at the main process. All of this is transparent to any customReporter
implementation, as only one instance of the customReporter
will be created to process the events, regardless of whether the tests run in same or forked process.Selector API to selectively run tests
New Framework API includes a set of comprehensive API to select tests for execution. Though new Framework API supports fine-grained test selection, current sbt's
test-only
andtest-quick
supports up to suite level selection only, orSuiteSelector
as defined in new Framework API. ThisFramework
implementation already supportsSuiteSelector
,NestedSuiteSelector
,TestSelector
andNestedTestSelector
, which should work once future sbt version supports them.Added new
Ignored
,Canceled
andPending
statusStatus
Ignored
,Canceled
andPending
are added to new Framework API, and they match perfectly with ScalaTest's ignored tests (now reported asIgnored
instead ofSkipped
), as well as canceled and pending tests newly added in ScalaTest 2.0.Added sbt Tagging support
Sbt supports task tagging, but has no support in old Framework API for test frameworks to integrate it. New Framework API supports it, and you can now use the following annotations to annotate your suite for sbt built-in resource tags:
CPU
Disk
Network
They will be mapped to corresponding resource tag
CPU
,Disk
andNetwork
in sbt.You can also define custom tag, which you'll need to write it as Java annotation:
which will be translated to
Tags.Tag("custom")
in sbt.