A Safety Net

I was to automate a testing process we were doing, where we basically had a checklist of items that needed to be validated, before we could sign off on a deployment for production. The list looked sort of like this:

1. Log in
2. Create a project and edit it
3. Upload a file
4. Create a view using the uploaded file
5. Validate a thumbnail is generated
6. Log into mobile application
7. Preview the new project
8. Delete the project
9. Log out

This kind of testing helps us make sure we don’t deploy silly mistakes to production. At a bare minimum, which is what this checklist represents, frequently used features should just work. Testing, in all its flavors, amounts to nothing if it’s not automated. After manually going through the checklist for a couple of weeks, it was clear we needed to start automating it.

Unit Tests are, of course, necessary to catch more subtle issues. Integration however aims for gross oversights, and generally being able to actually execute the application.

I gave wd a shot, paired with grunt-mocha-webdriver so that we could automate it using Grunt.

Grunt Automation

First off, we need to install the grunt task.

npm install --save-dev grunt-mocha-webdriver

Then, setting up the Grunt task was relatively easy.

mochaWebdriver: {
  options: {
    timeout: 1000 * 60,
    reporter: 'spec'
  },
  integration: {
    src: ['test/integration/**/*.js'],
    options: {
      usePhantom: true,
      usePromises: true
    }
  }
}

A couple of things to mention, in retrospect. At first we thought using Phantom, instead of a real browser, would be such a good idea, because it’d be faster to set up, and what not. A co-worker convinced me to use promises, and I went with it. The task could definitely use a better name, but at least it worked. So I had something to get me going.

There was one issue, though. I had to fire up the Node application myself, which meant extra work every time. I don’t like extra work, so I wrote a task that would start a Node process, run the tests, and then stop that process. I decided that I needed to wait on the application to start listening for requests, rather than shoving them to its face. I used a little port watching module which notifies me when an application starts listening on a given port, which worked just fine.

var app;

grunt.registerTask('integration-test-runner:start', function () {
  var done = this.async();
  var _ = require('lodash');
  var spawn = require('child_process').spawn;
  var finder = require('process-finder');
  var port = process.env.TEST_PORT || 3333;
  var watcher = finder.watch({ port: port, frequency: 400 });
  var env = _.clone(process.env);

  console.log('Spawning node process to listen on port %s...', port);

  env.PORT = port;
  app = spawn('node', ['app'], { stdio: 'inherit', env: env });

  process.on('exit', close);

  watcher.on('listen', function(pid) {
    console.log('Process %s listening on port %s...\nRunning tests.', pid, port)

    grunt.task.run(
      'mochaWebdriver:integration',
      'integration-test-runner:cleanup'
    );
    done();
  });
});

grunt.registerTask('integration-test-runner:cleanup', close);

function close () {
  if (app) {
    app.kill('SIGHUP');
    console.log('Process %s shutting down', app.pid);
  } else {
    console.log('Process not found');
  }
}

Note that grunt.task.run won’t actually run the task in place, but enqueue it so that it runs after the currently executing task, which is why done is called immediately afterwards. Giving the app an uncommon port was useful because I could run the application in port 3000, like it does by default, side-by-side with the automated tests. Lastly, the cleanup logic helped me avoid issues with the Node process taking over the port, preventing those infamous EADDRINUSE errors.

The First Test

Writing the first test was kind of awkward because I wasn’t really familiar with promises (did you know they’re coming to JavaScript in Harmony?), but I decided to give them a shot, anyway. Here is the first test, trying out the login logic.

var port = process.env.TEST_PORT || 3333;
var base = 'http://localhost:' + port;

it("handles an invalid login", function (done) {
  this.browser
    .get(base + '/admin')

    // assert we're taken to the login page
    .url().then(function (url) {
      console.log('Logging in...');
      return assert.equal(url, base + '/login');
    })

    // enter invalid credentials and submit
    .elementById('email').type('me@here.com')
    .elementById('pass').type('wayoff')
    .elementByTagName('form').submit();

    // assert we're still in the login page
    .url().then(function (url) {
      return assert.equal(url, base + '/login');
    })

    .then(done, done);
});

Looks really simple, innocent, and straightforward, right? I know! Thing is that actually getting it to work took a lot of effort, because the API is so underdocumented, I actually had to log Object.keys(this.browser) and go through the methods, trying to figure out which one did what I intended to do (submit the form, or type into an input). These are all symptoms of a lousy documentation. The API could be worse, as it at least attempts to mirror some of the methods found in the native DOM.

Context Barrier

Suppose you get past this initial barrier. Then there’s the context issues, and understanding how to transverse the DOM properly. If you want to get an element at random, navigate away, and then come back and select the same element, you’re gonna have a bad time.

This one, however, is mostly a matter of befriending promises and accumulating experience with the wd API. In particular, the way .then statements work is pretty cryptic. If they return an element, then that’ll be the context for the next promise in the chain, if they don’t then the browser object is used, and if you prepend '>' to selector requests, the context is used to restrict the query. For example:

this.browser
  .elementByCssSelector('.some-thing')
  .elementByCssSelector('>', '.some-thing-child')
  .then(function (element) {
    // return the element or you're back to the browser
    return element;
  });

Obviously, though, when you go for something like .text().then(function (value) {}), you’re pretty much screwed because you don’t have a reference to the element anymore, unless you’ve previously persisted it to a variable.

Capturing Page Load Errors

You’d think capturing page load errors is something that anyone would like to use. That is, logging errors that happen right after a request completes, during interpretation or execution of a piece of code. Well, if you google around, the only real solution to this problem is using client-side JavaScript, and patching onerror so that you can keep track of errors.

window.__wd_errors = [];
window.onerror = function (message, url, ln) {
  window.__wd_errors.push({
    message: message,
    url: url,
    ln: ln
  });
};

As WebDriver doesn’t really provide a mechanism to inject into the response stream, or manipulate it in any way (that I could find, anyways), you’re stuck with patching the application if a TEST_INTEGRATION environment variable or similar is turned on. On the testing side of things, you could augment the promise chain prototype to print the list of errors, after navigating to a page.

wd.PromiseChainWebdriver.prototype.throwIfJsErrors = function () {
  return this
    .safeEval('window.__wd_errors')
    .then(function (errors) {
      if (errors && errors.length) {
        console.log('Detected %s Error(s)', errors.length);

        errors.forEach(function (error) {
          console.log('%s\nAt %s, File %s', error.message, error.ln, error.url);
        });
      }
    });
};

This helped me catch an unexpected issue. Phantom doesn’t have Function.prototype.bind, and it won’t get included until 2.0.0, which doesn’t seem to be happenning any time soon. Temporarily, I added a polyfill for Function.prototype.bind to the file which had the the page load error capturing.

File Uploading is a Nightmare

The worst offender of all, was file uploading. Of course, documentation would’ve helped. It’s like nobody wants to talk about integration testing anyways, so googling around doesn’t do you a lot of good either. The best I could come up with was some information on a discussion on an issue on GitHub, and maybe questions about the WebDriver implementation in Java, which I attempted to mirror.

I tried everything. At first, I went with the API: find the element, then use .sendKeys(<path>). Nope, that won’t work. Okay, maybe it was just .type(<path>)? No. Need a .click() in between? Wrong again. You see, the lack of documentation makes it very hard for the consumer to know exactly how wrong their approach is. That represents a huge problem, because you’ll keep on trying things out blindly, hoping to eventually get it right. You won’t.

After lots of googling and a some desperate attempts, I stumbled upon a corner of the internet where I read that this functionality wasn’t implemented a few months ago, sure this pull request sounds like it should be working, but this issue on GhostDriver suggests otherwise, and I got tired of sifting through issue lists figuring out whether what I was trying to do was even supported.

Okay, great, so I decided to try something else. I know! I’m fine not testing the button click itself, that’s something a unit test could do. I care about the grand scheme of things. I need to upload the file, though. There’s no getting around that. Luckily, the page I was testing wasn’t submitting the form directly. It creates a FormData object, and places the files there, and then it sends that. All I need is to eval the right string, and it’ll all be over!

Some googling gave me the formula. Rather than give the file path to WebDriver, I had to hand over the Blob data directly to the browser. Converting the file to the correct format took a little trial and error, but the renewed spirit was there. Here’s a small addition to the wd promise chain prototype, so I could re-use my awesome file upload hack, some day.

wd.PromiseChainWebdriver.prototype.uploadSomething = function (file, scope) {
  console.log('Attempting file upload...');

  var mime = require('mime');
  var name = path.basename(file);
  var blob = fs.readFileSync(file, { encoding: 'base64' });
  var code = [
    util.format('var bytes = atob("%s");', blob),
    'var codepoints = Array.prototype.map.call(bytes, function (n) { return n.charCodeAt(0); });',
    'var data = new Uint8Array(codepoints);",
    util.format('var blob = new Blob([data], { type: "%s" });', mime.lookup(file)),
    util.format('blob.name = '%s';', name),
    'var formData = new FormData();',
    util.format('formData.append("%s", blob);'
  ].join(' ');

  return this
    .safeEval(code)
    .then(function () {
      console.log('File upload in progress.');
    });
};

Feeling great! Let’s do this! …nope, not working. Blob is unsupported in Phantom < 2.0.0, just like Function.prototype.bind. The polyfill I tried out didn’t work either, and I simply moved on to Chrome.

Moving to Chrome

Okay, fine. Rather than using the unreliable Ghost browser, I needed the real thing, and so I went with Chrome. To run tests with Chrome, I needed to change things up quite a bit. First off, I found a really simple selenium server installer which did the trick of firing up a Selenium server for me.

npm install -g selenium-standalone

Now I could fire up an instance in my command line. By the way, it requires Java!

start-selenium

Wait a minute… that’s too easy! Oh yeah, that’s right, grunt-mocha-webdriver doesn’t run against local selenium instances, even though a pull request, which adds that functionality, is already a month old. I went ahead and created a package out of the pull request, using that I could test against the local selenium instance created by start-selenium.

I really wanted to keep the selenium instance contained in the Grunt task, as well, so I went ahead and cloned selenium-standalone, adding a programmatic API. After that, it was just a matter of pulling it into a new Grunt task. That task would spawn a selenium server instance consuming the API I’ve just written.

var selenium = require('selenium-standalone-painful').start({
  stdio: 'pipe'
});

var ready = /Started org\.openqa\.jetty\.jetty\.Server/i;

selenium.stdout.on('data', function () {
  if (ready.test(data.toString())) {
    grunt.task.run('the-next-one');
  }
});

process.on('exit', function () {
  if (selenium) {
    selenium.kill('SIGHUP');
  }
});

That’s it! Then I decided to improve the reusability by pulling it out of its host project.

Introducing grunt-integration

I built a tool specifically to deal with the issues ordeal I went through while ramping up on my integration testing experience on Node applications. Concretely, these are the features I want in an integration testing module, and also the goals of grunt-integration:

• Start a local selenium server instance
• Start a local program, such as node application
• Wait for the program to listen on a specific port
• Execute integration tests using Mocha and WebDriver
• Using real browsers, such as Chrome
• Automatically, in one command
• Less painful installation process, please!

I’m considering adding some extensions to wd so that dealing with some of the issues I’ve described here is not so painful. The wd API could definitely get some love, but you can’t do a lot better than what it currently has. HTTP injection would be something that I’d love to see there, but I don’t think its even possible with Selenium.

You can check out the ongoing progress on GitHub, and maybe even collaborate with your expertise!