JsTestr
Overview
Jstestr is broken up into 3 independent pieces. There is testing framework which includes support for asynchronous tests using a callback or a future, test node creation, and full page testing, console logging and reporting. There is an assert module which includes asserts for equality, truthiness, object types, mock functions with argument verification, and more. Finally, there is synthetic events based GUI testing framework.
Setup
The framework makes use of AMD to load code and define dependencies. Any AMD compliant loader may be used. There are example runner HTML pages which configure RequireJS or dojo to load and execute tests. The example runners make some assumptions about the relative locations of the loader script files. If the assumptions are not correct the runner HTML page can be copied and configured correctly. For example, if a dojoConfig variable needs to be set before loading, the dojoRunner.html page can be modified to include that config script. The git submodules are not required and are only used for the framework tests and demos.
Test Framework
The test framework provides a way to define suites of tests. It also has provides an HTML page to run all the defined tests. There are two output adaptors, console and graphical. The graphical output includes a list of all the registered suites and tests as well as an output log area which is color coded. There are also buttons which can be used to execute individual tests or suites.
The test framework is available through the jstestr/test
module. It
provides a function to register a suite of tests called
defineSuite
. It accepts a suite name and an object defining the set
of tests. Here is a quick example of what a basic suite of tests might look
like:
define(["jstestr/test", "jstestr/assert"], function (test, assert) {
test.defineSuite("A Test Suite", {
"A Test Case": function () {
this.widget = new Widget();
this.widget.doSomething();
assert.isTrue(this.widget.prop, "Prop should be true");
},
"Another Test Case": function () {
this.widget = new Widget({foo: "abc123"});
this.widget.doSomethingElse();
assert.isEqual("abc123", this.widget.foo, "foo should be abc123");
}
});
});
Running Tests
There are a variety of ways to execute tests (browser, nodejs, phantomjs). In
a browser, a runner HTML page can be used to load the tests. There are example
runner pages in jstestr/runner
. Both dojo and RequireJS are
supported out of the box, but any AMD complient loader should work. The
browser loaders accepts a number of different query parameters which can be
used to control how the page loads and executes tests. It includes a copy of
RequireJS to support AMD module loading. The supported query parameters are:
module
- The AMD module to load which defines the tests.suite
- The name of the suite to execute.suites
- A pattern to use to find suites to execute (* as wild cards).test
- The suite and test name to execute, separated by a colon.config
- Extra RequireJS configuration (as JSON).
The runner page can be used to execute the test suite for JsTestr: jstestr/runner/runner.html?module=tests/testAll
Test Cases
Test cases are functions which can either execute synchronously or
asynchronously. Synchronous tests are considered successful if they complete
without throwing any exceptions. In the above example, the assert functions
throw errors if they do not pass. Tests can also be defined using an object with
a mandatory property named test
which must contain the actual test
function. A special property named timeout
can also be specified in
the test object which specifies how long to wait for an asynchronous test to
complete before failing it.
Asynchronous functions can be written in two ways. The first way is to return a
deferred or future. The future must contain a function named then
which accepts two callback parameters. The first callback must be executed if
the test is successful, and the second must be executed if the test is
successful. Note: if the test completes synchronously, but also returns a
future, it must execute the correct callback once the then
function
is called. Optionally, if the future provides a cancel
method it
will be executed if the test takes longer than the test timeout allows.
The second way to write an asynchronous test is to have the test function accept
a parameter that must be called done
. The done
parameter is a function which must be executed when the test is complete. If it
is passed true, the test is considered successful, if it is passed false, it is
considered failed. The done
function has a helper field called
wrap
which can be used to wrap functions which may throw errors. If
the function completes without error, the test is considered successful, if it
throws an error, it is considered failed.
test.defineSuite("Another Test Suite", {
"Synchronous Test": function () {
this.widget = new Widget();
this.widget.doSomething();
assert.isTrue(this.widget.prop, "Prop should be true");
},
"Synchronous Test With Object": {
widget: new Widget(),
otherProp: true,
test: function () {
this.widget.doSomething();
assert.isTrue(this.otherProp, "Other test property");
}
},
"Future Asynchronous Test": function () {
var future = new Future();
setTimeout(function () {
future.resolve();
}, 0);
return future;
},
"Done Asynchronous Test": function (done) {
setTimeout(function () {
done(true);
}, 0);
},
"Done With Asynchronous Test": function (done) {
setTimeout(done.with(function () {
assert.isTrue(true, "true should be true");
}), 0);
}
});
Ignoring Tests
Sometimes a test point needs to be ignored for some reason. Tests can be ignored
by prepending their name with //
. The test will still show up in the
list but will not execute. If a test only needs to be disabled in some situations
(on ie or something), then a condition can be added with //{condition}
.
test.defineSuite("Test Suite", {
"//Simple Ignored": function () {
assert.isTrue(false, "This will not run.");
},
"//{ie}Ignored in IE": function () {
assert.isFalse(test.conditions.ie, "This will not run in IE.");
},
"//{!ie}Not Ignored in IE": function () {
assert.isTrue(test.conditions.ie, "This will only run in IE.");
}
});
There is a default set of conditions available on the test.conditions
object:
{
msie: !!(global.attachEvent && !global.opera),
opera: !!global.opera,
webkit: navigator.userAgent.indexOf('AppleWebKit/') >= 0,
safari: navigator.userAgent.indexOf('AppleWebKit/') >= 0 &&
navigator.userAgent.indexOf('Chrome/') === -1,
gecko: navigator.userAgent.indexOf('Gecko') >= 0,
mobileSafari: !! navigator.userAgent.match(/Apple.*Mobile.*Safari/)
}
Generated Tests
Tests are sometimes repetative series of asserts with a range of input values.
A set of tests can be generated based on arrays of input values specified in
a parameters
field of the test:
test.defineSuite("Generated Tests", {
"Test $name": {
parameters {
name: ["One", "Two", "Three"],
a: [1, 2, 3],
b: [3, 2, 1],
expected: [4, 4, 4]
},
test: function (a, b, expected) {
assert.equal(expected, a + b, "a plus b");
}
}
});
The example test definition will create 3 test cases with the inputs from the
parameters
map. The name of the tests can be generated based on
any of the fields of the parameters
map using string replace.
Special Suite Function
There are a few special fields that can be defined in the suite object:
beforeEach
- Executed before each test case.afterEach
- Executed after each test case (successful or not).beforeSuite
- Executed at the start of the suite.afterSuite
- Executed after the suite is done.pageUnderTest
- Page to load before the suite is started.
beforeEach/afterEach
These functions are executed before and after each test. They are executed with
the same context as the test, so values can be shared between the functions
using this
. The afterEach
function is execute after
the test is done event if the test failed. If either function throws an error,
the test is considered failed. These functions must be synchronous.
beforeSuite/afterSuite
These functions are executed before the first test in the suite and after the
last test in the suite. These can be useful for things which are either
expensive to construct or don't need to be cleaned up between tests, but the
beforeEach
/afterEach
functions should be preferred.
The context used by the functions is available in the test functions through the
suite
property of the test's context. These functions must be
synchronous.
pageUnderTest
The pageUnderTest
property can be used to specify a URL which will
be loaded before the first test is executed. The page is loaded into an iFrame
and the iFrame's contentWindow
and contentDocument
properties are passed to the tests in the fields document
and
global
of the function's context. The iFrame is automatically
destroyed when the suite finishes, but it is not reset between each test. The
framework will wait up to 20 seconds for the page to load.
test.defineSuite("Page Under Test", {
"pageUnderTest": require.toUrl("./pageUnderTest.html"),
"Page Load": function () {
assert.isTrue(this.document, "Document should be set");
assert.isTrue(this.global, "Global should be set");
}
});
Asserts
The assert module provides a number of assertion functions as well as mock
functions. The module is available in jstestr/assert
. All methods
take an optional help message to add to the error if the assertion fails. The
assert module provides the following methods:
isTrue(value, help)
- Test for truthy value.isFalse(value, help)
- Test for falsy value.isEqual(expected, actual, help)
- Test for deep equality.isNotEqual(expected, actual, help)
- Test for deep inequality.isSame(expected, actual, help)
- Test for strict equality (===).isNotSame(expected, actual, help)
- Test for strict inequality (!==).isClose(expected, actual, toleranceOrHelp, help)
- Test that the actual number is close to the expected number within some tolerance.isObject(value, help)
- Throws if the value is not an object. Arrays are not counted as objects.isArray(value, help)
- Test for arrays. Checks the constructor, so it must be an actual Array, not a pseudo array.isFunction(value, help)
- Test for functions.isNumber(value, help)
- Test for numbers.isString(value, help)
- Test for strings.doesThrow(expectedType, func, help)
- Call the function and verify that it throws an error of the expected type.matches(expected, actual, help)
- Test that the fields and values in the expected object match the actual value. See the section below for details.createMockFunction():Function
- Create a mock function which can verify how many times it is called and the arguments passed to it.createMockObject(methodsOrObj):Object
- Create an object with a set of mock methods. Each method is a mock function.
Matches
The assert.matches
function matches the actual value against an
expected pattern. The expected pattern can have fewer fields, use the
assert.any
, assert.range(low, high)
, and regular
expressions to check the object.
assert.matches(assert.range(1, 2), 1.5, "Number in range of 1:2");
assert.matches(/a regex/, "a regex should match", "String matching");
assert.matches({a: 1, b: 2}, {a: 1, b: 2, c: "extra"}, "Extra fields are ignored");
assert.matches({a: assert.any, b: 2}, {a: "anything", b: 2}, "Field can be anything but undefined");
Mocks
Mocks are a way to isolate components from the rest of the system. They are useful in a number of different ways. The assert module provides two ways to use mocks. First, a function can be created which can be configured using the following methods:
times(num)
- Set the mock to expect to be called the specified number of times.expect(arg1, arg2, ...)
- Expect the mock the be called with the specified arguments.result(value)
- Return the specified result when the function is called.error(err)
- Throw the specified error object when the function is called.verify(help)
- Verify the number of times the function has been called and the arguments used.reset()
- Reset the mock's internal count of how it's been called. This will not reset the expected arguments, results, errors, or called times.
var mock = assert.createMockFunction();
mock.expect(1, "two", {three: true}).result(4);
mock.times(2);
var result1 = mock(1, "two", {three: true});
var result2 = mock(1, "two", {three: true});
mock.verify("Arguments should be: 1, "two", {three: true}");
assert.isEqual(4, result1, "Result 1 should be 4");
assert.isEqual(4, result2, "Result 2 should be 4");
Mock objects are simply collections of mock functions which can be verified and
reset as a group (verify
, reset
). The methods are
created based on either an array of strings, which represent the names of
methods to mock, or an object containing methods to be mocked.
var mock = assert.createMockObject(["method1", "method2"]);
mock.method1();
mock.method2();
mock.verify();
var Constructor = function () {};
Constructor.prototype.method1 = function () {};
Constructor.prototype.method2 = function () {};
Constructor.prototype.prop1 = true;
// mock from a constructor function
mock = assert.createMockObject(Constructor);
mock.method1();
mock.method2();
assert.isFalse(mock.prop1, "This should be undefined");
mock.verify();
Synthetic Events
The synthetic events module provides functionality for simulating mouse operations, keyboard events such as typing, focus changes. It also provides a way to asynchronously query the DOM.
The tests are executed asynchronously using a queue of tasks. The Synthetic
event module is instantiated and each method call will simply add to the queue.
The queue must be started before any tasks will be executed. The
start
method returns a cancelable future object which can be
directly returned to the test framework.
Here is an example of a basic synthetic event test:
test.defineSuite("Synthetic Test", {
"pageUnderTest": require.toUrl("./testPage.html"),
"Click Button": function () {
var synth = new Synthetic({document: this.document, global: this.global});
synth.click("#aButton");
return synth.start();
}
});
The Synthetic constructor function accepts options for overriding the
document
and global
object used while executing the
tasks. Queries and other DOM interaction will go against the specified objects.
All methods accept an optional handler
function, which will be
executed when the task completes successfully, and optional options object,
which is used for options such as the shift key flag. Any place an element is
passed as a parameter a selector can be passed instead. The first element which
matches the selector will be used.
The Synthetic event class provides the following methods:
byId(id, handler, options)
- Search the DOM for an element id.query(selector, handler, options)
- Query for a single element based on a selector. If multiple elements match, the first one will be used.queryAll(selector, expectedCount, handler, options)
- Query for the specified number of elements which match the selector.type(string, element, handler, options)
- Type into the specified element.click(element, handler, options)
- Click the specified element.doubleClick(element, handler, options)
- Double click the specified element.hover(element, handler, options)
- Hover over the specified element.move(from, to, handler, options)
- Move the mouse from one place to another.drag(from, to, handler, options)
- Drag the mouse from one place to another. The difference frommove
is that it starts with a mouse down and ends with mouse up.
byId/query/queryAll
The trio of DOM query methods can be used to lookup a specific element so that
it can be manipulated in some way. They can also be used as assertions. They
poll the DOM for a certain amount of time (overridable in the
options
object) until the element(s) are found. If they are not
found the test is failed.
The queryAll
method's expectedCount
parameter can be
specified as either a number (including 0) or a string. If it is a string, then
it must be a combination of comparison expression involving
<, >, =, &, |
and number constants. The comparisons are evaluated
left-to-right and all white space is ignored. Here are a few examples:
"<3"
- Less than 3 elements.">2"
- More than 2 elements."=0"
- Exactly 0 elements."<5 & >1 | =10"
- Less than 5 and more than 1 or exactly 10.
type
The type
method is used to type characters and keys into an element.
Printing characters are specified as characters in the string. Other keys can be
typed by using special codes inside the strings such as [backspace]
.
Not all other keys are supported. The following are minimally supported:
[backspace]
- Key code: 8.[escape]
- Key code: 27.\n
- Key identifier:Enter
, key code: 10.[left]
- Key identifier:Left
, key code: 37.[up]
- Key identifier:Up
, key code: 38.[right]
- Key identifier:Right
, key code: 39.[down]
- Key identifier:Down
, key code: 40.[pageup]
- Key identifier:PageUp
, key code: 33.[pagedown]
- Key identifier:PageDown
, key code: 34.
sequence.type("this is a strig[backspace]ng", "#anElement", function (element) {
console.log("Typing completed on element:", element);
});
click/doubleClick/hover
These three mouse related methods operate on single elements. click
and doubleClick
dispatch mousedown
,
mouseup
, and click
events. doubleClick
dispatches an extra dblclick
event after two clicks.
hover
dispatches mouseover
, mousemove
,
and then waits for the specified amount of time (default of 500ms).
sequence.click("#anElement", function (element) {
console.log("Click completed on element:", element);
});
move/drag
The move
and drag
methods take two elements as inputs
which represent the start and end points of a series of mousemove
events. As the mouse moves over elements in the page mouseout
and
mouseover
events are dispatched as appropriate. For the
drag
method, the sequence is started with a mousedown
and ended with mouseup
.
sequence.drag("#firstElement", ".secondElement", function (elementA, elementB) {
console.log("Drag completed:", elementA, elementB);
});