ECMAScript 6–otherwise known as "Harmony"–packs a plethora of new exciting features for Javascript. Several browsers have been sneaking in these additions under the hood for quite some time as specifications have reached consensus. Most of these language updates aren't anything unique to Javascript–in fact, most of the syntax and APIs closely mirror other mature languages and preprocessors like Python, Ruby, and CoffeeScript. The most impressive support so far comes from Firefox, whose Javascript engine SpiderMonkey is leaps and bounds ahead of the competition. In this post we'll benchmark the Harmony-way, and the "old way" with just a handful of these new language additions.

Predictions

Before even beginning this post, my attempts to search for similar performance studies turned up very little information. However, I figured it would be safe to predict that in cases where it was possible to implement a Harmony feature using pre-Harmony syntax, the older syntax would win. But, I think it's important to realize that language features aren't always centered around performance, but rather important concepts such as productivity, readability, portability, and simplicity. That said, I do expect the new language features to at least be comparable in performance. One of my favorite lines from the Zen of Python is "There should be one–and preferably only one–obvious way to do it." I believe that a lot of these new language additions help bring Javascript closer to that ideal.

The Test Setup

I'm not as concerned here with running tests in the most performant way, but rather running them in a consistent way so that reasonable conclusions can be drawn. As SpiderMonkey is the most accomplished Javascript engine in terms of Harmony support, I decided to run all tests against the latest github source. By not running tests within Firefox itself, I hoped to avoid whatever overhead might come from various DOM and browser specific APIs. I'll be using the Gulp.js task runner, which I discussed in last month's post, to execute the actual tests and collect results. For certain tests, to produce the non-Harmony syntax, I'll be relying on Google's traceur compiler . Each test will be ran one at a time, in their own SpiderMonkey process. As far as hardware is concerned, I'm running on a MacBook Air with OS X 10.9.2 from mid 2013 with a 1.7GHz Intel Core i7 processor and 8GB of RAM.

A Look at the Test Runner

I've pieced together a very simple stream for running tests and collecting results that can integrate with Gulp.js. The plugin basically works like this:

  1. Spawn a SpiderMonkey process with a Javascript test file
  2. Collect execution time information using Node's process.hrtime()
  3. Store results in a map
  4. Repeat steps 1 thru 3 till all tests are run
  5. Output results to a JSON file

I chose to use process.hrtime() from Node partly for convenience, but also because I noticed very little difference between it and SpiderMonkey's dateNow() and the system time command.

Here's a look at the plugin itself:

var fs = require('fs');
var through = require('through2');
var _ = require('lodash');
var spawn = require('child_process').spawn;
var gutil = require('gulp-util');
var PluginError = gutil.PluginError;


var Runner = function(opts){
    // default options
    opts = _.extend({
        output: './results.json',
        command: 'js24',
        args: ['-f']
    }, opts || {});

    // results map
    var results = {};

    return through.obj(function(file, enc, callback){
        // push null files or directories through
        if (file.isNull() || file.isDirectory()) {
            this.push(file);
            return callback();
        }

        var that = this;
        var start = process.hrtime();
        var sm = spawn(opts.command, opts.args.concat([file.path]));

        // emit any errorcs encountered by process
        sm.stderr.on('data', function(data) {
            that.emit('error', new PluginError({
                plugin: 'Runner',
                message: 'Spidermonkey error: ' + data
            }));
            callback();
        });

        // save results on process close
        sm.on('close', function(){
            var end = process.hrtime(start);
            results[file.path] = end[1] / 1000000;
            callback();
        });

    }, function(callback){
        // dump JSON to output file
        fs.writeFile(opts.output, JSON.stringify(results, null, 4), function(){
            callback();
        });
    });
};

module.exports = Runner;

Test Results

In the tests outlined below, I've attempted to use new syntax related features in a meaningful context. There are some items I'm skipping over despite them being available:

  • Promises - There's tons of amazing libraries–like Q–for doing this today, and I don't see how there could be any notable performance differences. Also, I struggle to see a good approach to performance testing here (I'm thinking memory usage might make more sense instead of speed). Promises have simply become such a necessary tool in the asynchronous world of Javascript that they're getting a special place in the standard API.
  • Spread operator and rest parameters - Both of these features are really just adding some syntax sugar to something that's already possible (just verbose and sometimes awkward) in Javascript. For example, you can simply use Function.prototype.apply() in place of the spread operator, and do various array slicing operations on function arguments to accomplish what the rest parameter does. They're not pretty, but they work.
  • Proxy -This one I'd actually like to test, but I feel like the way you'd interact with the proxy "equivalent" in pre-harmony would be too different. One of the most powerful use cases for a proxy object (in my opinion) comes when working with "virtual objects." You can pass a proxy object around throughout your application, and even if references are stored to it you can still change what that proxy is "pointing" to at any time. A simple example:

    function virtualProxyFactory(t){
        var target = t;
        return [
            {
                set: function(t){
                    return target = t;
                },
    
    
                get: function(){
                    return target;
                }
            },
            // we'll just implement getter and setter
            Proxy({}, {
                get: function(t, name, receiver){
                    return target[name];
                },
    
    
                set: function(t, name, val, receiver){
                    target[name] = val;
                }
            })
        ];
    }
    
    
    var doug = {
        name: 'Douglas'
    };
    var userProxyDescriptor;
    var userProxy;
    
    
    // yay destructuring!
    [userProxyDescriptor, userProxy] = virtualProxyFactory(doug);
    
    
    // pass proxied user to a view
    var view = new View({
        user: userProxy
    });
    
    
    // view.user.name === 'Douglas'
    
    
    // later we decide to change the user
    var brendan = {
        name: 'Brendan'
    };
    userProxyDescriptor.set(brendan);
    
    
    // .. now view.user.name === 'Brendan'
    

Parsing REST API Responses: Destructuring

Data processing is a common task in any web application. Using the Google API explorer, I've queried their Books API for "javascript" and saved the JSON response containing 10 books. In this test I'm simply extracting some nested property values from each book using object destructuring, and then returning a formatted string. The processing function that iterates over the book list is called 1000 times.

  • Pre-Harmony ≈ 25.9ms
  • Harmony ≈ 27.2ms (0.05x slower)

These results are too close to really declare a winner. Though destructuring can help you avoid things like creating temporary variables, it can also turn ugly fast (like in this example) when you're extracting data from complex objects. I would imagine once this feature goes mainstream, we'll start to see more variety in return types (namely in the form of arrays) from functions since they can be easily unpacked into local variables.

Overloaded API Methods: Default Params

A lot of popular Javascript libraries (like jQuery) provide API methods with optional arguments, many of which have some kind of sensible default. In this test, I've created a simple API object with three methods. Each method has only a few required arguments, followed by several optional arguments. The main API method accepts a particularly large number of arguments, as it's real role is to delegate operations to the other two API methods. Similar to the last test, I'll make 1000 calls with a fixed set of arguments.

  • Pre-Harmony ≈ 20.7ms
  • Harmony ≈ 21.0ms (0.01x slower)

Once again, the results are more or less the same. Default parameters are yet another feature we should feel safe using without any sort of performance penalty.

Matrix Multiplication: Generators, For…Of Loops and Array Comprehensions

MATMUL is a common performance test that makes use of embedded loops. This was a perfect opportunity to use features relating to iteration. I stumbled across this language benchmark suite which had some vanilla Javascript for MATMUL ready to go. I created a Harmony version where I used a few simple generators and nested list comprehensions to replace several layers of for loops. In the end–because I had to write a few custom generators–I didn't actually save many lines of code. My results from using two 100x100 matrices:

  • Pre-Harmony ≈ 22.3ms
  • Harmony ≈ 927.5ms (41x slower)

I must say, I was pretty super disappointed by this one. These are easily some of the most important new language features in Harmony. As a Pythonista, I've come to expect performance from list comprehensions and generators. I'm also a bit frustrated with Mozilla that they'd release features like this where performance is still such a problem. Perhaps the slower rollout of features in V8 is because they're being more meticulously implemented?

In an attempt to speed things up I ran this test again without array comprehensions, and instead only generators and for…of loops. Unfortunately, this still resulted in Harmony syntax being 26x slower. Please fix?

Conclusion

Harmony packs some powerful new additions to Javascript. Though some of these additions are still in need of polishing performance-wise (as demonstrated in the matmul test), it's still exciting to see Javascript's feature set evolving to that of other popular dynamic languages. I look forward to re-visiting the topic of performance in the near future once some other substantial language specifications (like classes and modules) reach draft state and start appearing in Javascript engines. For now, check out the tests on github and run them for yourself!

comments powered by Disqus