Framework

This class is ScalaTest's implementation of the new Framework API that is supported in sbt 0.13.

To use ScalaTest in sbt, you should add ScalaTest as dependency in your sbt build file, the following shows an example for using ScalaTest 2.0 with Scala 2.10.x project:

"org.scalatest" % "scalatest_2.10" % "2.0" % "test"

To pass argument to ScalaTest from sbt, you can use testOptions:

testOptions in Test += Tests.Argument("-h", "target/html")  // Use HtmlReporter

If you are using multiple testing frameworks, you can pass arguments specific to ScalaTest only:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/html") // Use HtmlReporter

Supported arguments

Integration in sbt 0.13 supports same argument format as Runner, except the following arguments:

• -p, -R -- runpath is not supported because test path and discovery is handled by sbt
• -s -- suite is not supported because sbt's test-only serves the similar purpose
• -A -- again is not supported because sbt's test-quick serves the similar purpose
• -j -- junit is not supported because in sbt different test framework should be supported by its corresponding Framework implementation
• -b -- testng is not supported because in sbt different test framework should be supported by its corresponding Framework implementation
• -c, -P -- concurrent/parallel is not supported because parallel execution is controlled by sbt.
• -q is not supported because test discovery should be handled by sbt, and sbt's test-only or test filter serves the similar purpose
• -T is not supported because correct ordering of text output is handled by sbt
• -g is not supported because current Graphic Reporter implementation works differently than standard reporter

New Features of New Framework API

New Framework API supports a number of new features that ScalaTest has utilized to support a better testing experience in sbt. The followings are summary of new features supported by the new Framework API:

• Specified behavior of single instance of Runner per project run (non-fork), and a new done method
• API to return nested tasks
• API to support test execution in fork mode
• Selector API to selectively run tests
• Added new Ignored, Canceled and Pending status
• Added sbt Tagging support

Specified behavior of single instance of Runner per project run (non-fork), and a new done method

In new Framework API, it is now a specified behavior that Framework's runner method will be called to get a Runner instance once per project run. Arguments will be passed when calling Framework's runner and this gives ScalaTest a good place to perform setup tasks, such as initializing Reporters.

There's also a new done on Runner interface, which in turns provide a good spot for ScalaTest to perform cleanup tasks, such as disposing the Reporters. HtmlReporter depends on this behavior to generate its index.html. In addition, done can return framework-specific summary text for sbt to render at the end of the project run, which allows ScalaTest to return its own summary text.

API to return nested tasks

In sbt version before 0.13, ScalaTest's nested suites are always executed sequentially regardless of parallelExecution value. In new Framework API, a new concept of Task is introduced, which its execute method can return more (nested) Tasks for execution. When parallelExecution is set to true (the default), sbt will execute the nested tasks in parallel.

Each Suite in ScalaTest now maps to a Task in sbt, and its nested suites are returned as nested Tasks. This enables parallel execution of nested suites in sbt.

API to support test execution in fork mode

Forking was added to sbt since version 0.12, you can find documentation for forking support in sbt at Forking in sbt.

Although forking is already available in sbt since 0.12, there's no support in old Framework API, until it is added in new Framework API that is supported in sbt 0.13. With API provided with new Framework API, ScalaTest creates real Reporters in the main process, and uses SocketReporter in forked process to send events back to the main process, and get processed by real Reporters at the main process. All of this is transparent to any custom Reporter implementation, as only one instance of the custom Reporter will be created to process the events, regardless of whether the tests run in same or forked process.

Selector API to selectively run tests

New Framework API includes a set of comprehensive API to select tests for execution. Though new Framework API supports fine-grained test selection, current sbt's test-only and test-quick supports up to suite level selection only, or SuiteSelector as defined in new Framework API. This Framework implementation already supports SuiteSelector, NestedSuiteSelector, TestSelector and NestedTestSelector, which should work once future sbt version supports them.

Added new Ignored, Canceled and Pending status

Status Ignored, Canceled and Pending are added to new Framework API, and they match perfectly with ScalaTest's ignored tests (now reported as Ignored instead of Skipped), as well as canceled and pending tests newly added in ScalaTest 2.0.

Added sbt Tagging support

Sbt supports task tagging, but has no support in old Framework API for test frameworks to integrate it. New Framework API supports it, and you can now use the following annotations to annotate your suite for sbt built-in resource tags:

• CPU
• Disk
• Network

They will be mapped to corresponding resource tag CPU, Disk and Network in sbt.

You can also define custom tag, which you'll need to write it as Java annotation:

import java.lang.annotation.Target;
import java.lang.annotation.Retention;
import org.scalatest.TagAnnotation;

@TagAnnotation("custom")
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE})
public @interface Custom {}

which will be translated to Tags.Tag("custom") in sbt.