JsTestr

JavaScript testing framework with synthetic event based GUI testing support as well as asynchronous tests.

Overview

Jstestr is broken up into 3 independent pieces. There is testing framework which includes support for asynchronous tests using a callback or a future, test node creation, and full page testing, console logging and reporting. There is an assert module which includes asserts for equality, truthiness, object types, mock functions with argument verification, and more. Finally, there is synthetic events based GUI testing framework.

  1. Setup
  2. Test Framework
  3. Running Tests
  4. Test Cases
  5. Ignoring Tests
  6. Generated Tests
  7. Special Suite Functions
  8. Asserts
  9. Mocks
  10. Synthetic Events

Setup

The framework makes use of AMD to load code and define dependencies. Any AMD compliant loader may be used. There are example runner HTML pages which configure RequireJS or dojo to load and execute tests. The example runners make some assumptions about the relative locations of the loader script files. If the assumptions are not correct the runner HTML page can be copied and configured correctly. For example, if a dojoConfig variable needs to be set before loading, the dojoRunner.html page can be modified to include that config script. The git submodules are not required and are only used for the framework tests and demos.

Test Framework

The test framework provides a way to define suites of tests. It also has provides an HTML page to run all the defined tests. There are two output adaptors, console and graphical. The graphical output includes a list of all the registered suites and tests as well as an output log area which is color coded. There are also buttons which can be used to execute individual tests or suites.

The test framework is available through the jstestr/test module. It provides a function to register a suite of tests called defineSuite. It accepts a suite name and an object defining the set of tests. Here is a quick example of what a basic suite of tests might look like:


define(["jstestr/test", "jstestr/assert"], function (test, assert) {
    test.defineSuite("A Test Suite", {
        "A Test Case": function () {
            this.widget = new Widget();
            this.widget.doSomething();
            assert.isTrue(this.widget.prop, "Prop should be true");
        },

        "Another Test Case": function () {
            this.widget = new Widget({foo: "abc123"});
            this.widget.doSomethingElse();
            assert.isEqual("abc123", this.widget.foo, "foo should be abc123");
        }
    });
});
                

Running Tests

There are a variety of ways to execute tests (browser, nodejs, phantomjs). In a browser, a runner HTML page can be used to load the tests. There are example runner pages in jstestr/runner. Both dojo and RequireJS are supported out of the box, but any AMD complient loader should work. The browser loaders accepts a number of different query parameters which can be used to control how the page loads and executes tests. It includes a copy of RequireJS to support AMD module loading. The supported query parameters are:

The runner page can be used to execute the test suite for JsTestr: jstestr/runner/runner.html?module=tests/testAll

Test Cases

Test cases are functions which can either execute synchronously or asynchronously. Synchronous tests are considered successful if they complete without throwing any exceptions. In the above example, the assert functions throw errors if they do not pass. Tests can also be defined using an object with a mandatory property named test which must contain the actual test function. A special property named timeout can also be specified in the test object which specifies how long to wait for an asynchronous test to complete before failing it.

Asynchronous functions can be written in two ways. The first way is to return a deferred or future. The future must contain a function named then which accepts two callback parameters. The first callback must be executed if the test is successful, and the second must be executed if the test is successful. Note: if the test completes synchronously, but also returns a future, it must execute the correct callback once the then function is called. Optionally, if the future provides a cancel method it will be executed if the test takes longer than the test timeout allows.

The second way to write an asynchronous test is to have the test function accept a parameter that must be called done. The done parameter is a function which must be executed when the test is complete. If it is passed true, the test is considered successful, if it is passed false, it is considered failed. The done function has a helper field called wrap which can be used to wrap functions which may throw errors. If the function completes without error, the test is considered successful, if it throws an error, it is considered failed.


test.defineSuite("Another Test Suite", {
    "Synchronous Test": function () {
        this.widget = new Widget();
        this.widget.doSomething();
        assert.isTrue(this.widget.prop, "Prop should be true");
    },

    "Synchronous Test With Object": {
        widget: new Widget(),
        otherProp: true,
        test: function () {
            this.widget.doSomething();
            assert.isTrue(this.otherProp, "Other test property");
        }
    },

    "Future Asynchronous Test": function () {
        var future = new Future();
        setTimeout(function () {
            future.resolve();
        }, 0);
        return future;
    },

    "Done Asynchronous Test": function (done) {
        setTimeout(function () {
            done(true);
        }, 0);
    },

    "Done With Asynchronous Test": function (done) {
        setTimeout(done.with(function () {
            assert.isTrue(true, "true should be true");
        }), 0);
    }
});
                

Ignoring Tests

Sometimes a test point needs to be ignored for some reason. Tests can be ignored by prepending their name with //. The test will still show up in the list but will not execute. If a test only needs to be disabled in some situations (on ie or something), then a condition can be added with //{condition}.


test.defineSuite("Test Suite", {
    "//Simple Ignored": function () {
        assert.isTrue(false, "This will not run.");
    },

    "//{ie}Ignored in IE": function () {
        assert.isFalse(test.conditions.ie, "This will not run in IE.");
    },

    "//{!ie}Not Ignored in IE": function () {
        assert.isTrue(test.conditions.ie, "This will only run in IE.");
    }
});
                

There is a default set of conditions available on the test.conditions object:


{
    msie: !!(global.attachEvent && !global.opera),
    opera: !!global.opera,
    webkit: navigator.userAgent.indexOf('AppleWebKit/') >= 0,
    safari: navigator.userAgent.indexOf('AppleWebKit/') >= 0 &&
            navigator.userAgent.indexOf('Chrome/') === -1,
    gecko: navigator.userAgent.indexOf('Gecko') >= 0,
    mobileSafari: !! navigator.userAgent.match(/Apple.*Mobile.*Safari/)
}
                

Generated Tests

Tests are sometimes repetative series of asserts with a range of input values. A set of tests can be generated based on arrays of input values specified in a parameters field of the test:


test.defineSuite("Generated Tests", {
    "Test $name": {
        parameters {
            name: ["One", "Two", "Three"],
            a: [1, 2, 3],
            b: [3, 2, 1],
            expected: [4, 4, 4]
        },
        test: function (a, b, expected) {
            assert.equal(expected, a + b, "a plus b");
        }
    }
});
                

The example test definition will create 3 test cases with the inputs from the parameters map. The name of the tests can be generated based on any of the fields of the parameters map using string replace.

Special Suite Function

There are a few special fields that can be defined in the suite object:

beforeEach/afterEach

These functions are executed before and after each test. They are executed with the same context as the test, so values can be shared between the functions using this. The afterEach function is execute after the test is done event if the test failed. If either function throws an error, the test is considered failed. These functions must be synchronous.

beforeSuite/afterSuite

These functions are executed before the first test in the suite and after the last test in the suite. These can be useful for things which are either expensive to construct or don't need to be cleaned up between tests, but the beforeEach/afterEach functions should be preferred. The context used by the functions is available in the test functions through the suite property of the test's context. These functions must be synchronous.

pageUnderTest

The pageUnderTest property can be used to specify a URL which will be loaded before the first test is executed. The page is loaded into an iFrame and the iFrame's contentWindow and contentDocument properties are passed to the tests in the fields document and global of the function's context. The iFrame is automatically destroyed when the suite finishes, but it is not reset between each test. The framework will wait up to 20 seconds for the page to load.


test.defineSuite("Page Under Test", {
    "pageUnderTest": require.toUrl("./pageUnderTest.html"),

    "Page Load": function () {
        assert.isTrue(this.document, "Document should be set");
        assert.isTrue(this.global, "Global should be set");
    }
});
                

Asserts

The assert module provides a number of assertion functions as well as mock functions. The module is available in jstestr/assert. All methods take an optional help message to add to the error if the assertion fails. The assert module provides the following methods:

Matches

The assert.matches function matches the actual value against an expected pattern. The expected pattern can have fewer fields, use the assert.any, assert.range(low, high), and regular expressions to check the object.


assert.matches(assert.range(1, 2), 1.5, "Number in range of 1:2");
assert.matches(/a regex/, "a regex should match", "String matching");
assert.matches({a: 1, b: 2}, {a: 1, b: 2, c: "extra"}, "Extra fields are ignored");
assert.matches({a: assert.any, b: 2}, {a: "anything", b: 2}, "Field can be anything but undefined");
                

Mocks

Mocks are a way to isolate components from the rest of the system. They are useful in a number of different ways. The assert module provides two ways to use mocks. First, a function can be created which can be configured using the following methods:


var mock = assert.createMockFunction();
mock.expect(1, "two", {three: true}).result(4);
mock.times(2);

var result1 = mock(1, "two", {three: true});
var result2 = mock(1, "two", {three: true});
mock.verify("Arguments should be: 1, "two", {three: true}");
assert.isEqual(4, result1, "Result 1 should be 4");
assert.isEqual(4, result2, "Result 2 should be 4");
                

Mock objects are simply collections of mock functions which can be verified and reset as a group (verify, reset). The methods are created based on either an array of strings, which represent the names of methods to mock, or an object containing methods to be mocked.


var mock = assert.createMockObject(["method1", "method2"]);
mock.method1();
mock.method2();
mock.verify();

var Constructor = function () {};
Constructor.prototype.method1 = function () {};
Constructor.prototype.method2 = function () {};
Constructor.prototype.prop1 = true;

// mock from a constructor function
mock = assert.createMockObject(Constructor);
mock.method1();
mock.method2();
assert.isFalse(mock.prop1, "This should be undefined");
mock.verify();
                

Synthetic Events

The synthetic events module provides functionality for simulating mouse operations, keyboard events such as typing, focus changes. It also provides a way to asynchronously query the DOM.

The tests are executed asynchronously using a queue of tasks. The Synthetic event module is instantiated and each method call will simply add to the queue. The queue must be started before any tasks will be executed. The start method returns a cancelable future object which can be directly returned to the test framework.

Here is an example of a basic synthetic event test:


test.defineSuite("Synthetic Test", {
    "pageUnderTest": require.toUrl("./testPage.html"),

    "Click Button": function () {
        var synth = new Synthetic({document: this.document, global: this.global});
        synth.click("#aButton");
        return synth.start();
    }
});
                

The Synthetic constructor function accepts options for overriding the document and global object used while executing the tasks. Queries and other DOM interaction will go against the specified objects.

All methods accept an optional handler function, which will be executed when the task completes successfully, and optional options object, which is used for options such as the shift key flag. Any place an element is passed as a parameter a selector can be passed instead. The first element which matches the selector will be used.

The Synthetic event class provides the following methods:

byId/query/queryAll

The trio of DOM query methods can be used to lookup a specific element so that it can be manipulated in some way. They can also be used as assertions. They poll the DOM for a certain amount of time (overridable in the options object) until the element(s) are found. If they are not found the test is failed.

The queryAll method's expectedCount parameter can be specified as either a number (including 0) or a string. If it is a string, then it must be a combination of comparison expression involving <, >, =, &, | and number constants. The comparisons are evaluated left-to-right and all white space is ignored. Here are a few examples:

type

The type method is used to type characters and keys into an element. Printing characters are specified as characters in the string. Other keys can be typed by using special codes inside the strings such as [backspace]. Not all other keys are supported. The following are minimally supported:


sequence.type("this is a strig[backspace]ng", "#anElement", function (element) {
    console.log("Typing completed on element:", element);
});
                

click/doubleClick/hover

These three mouse related methods operate on single elements. click and doubleClick dispatch mousedown, mouseup, and click events. doubleClick dispatches an extra dblclick event after two clicks. hover dispatches mouseover, mousemove, and then waits for the specified amount of time (default of 500ms).


sequence.click("#anElement", function (element) {
    console.log("Click completed on element:", element);
});
                

move/drag

The move and drag methods take two elements as inputs which represent the start and end points of a series of mousemove events. As the mouse moves over elements in the page mouseout and mouseover events are dispatched as appropriate. For the drag method, the sequence is started with a mousedown and ended with mouseup.


sequence.drag("#firstElement", ".secondElement", function (elementA, elementB) {
    console.log("Drag completed:", elementA, elementB);
});