Thursday, June 28, 2012

careful with synchronous operations in async iterator

We use async flow control Node.JS library a lot at work. It provides various convenient functions to guide you through async programming mess. But lately, we found an interesting issue/lesson using async library:

Better avoid using synchronous operations inside the iterator, otherwise, when the number of items to iterate is big enough, you will exceed the call stack size.

For example, this code snippet just gives the basic idea (although realistically, you don't really need async to output a large array ;-)

var async = require('async');
var a = [];

for(var i = 0; i < 3040000; i++) {
 a.push(i);
}

async.forEachSeries(a,
 function (item, cb){
  console.log(item); // non-async operation
  cb();
 },
 function () {
  console.log('all done');
 }
);


When you run it, you will get:

0
1
2
...
RangeError: Maximum call stack size exceeded

The problem here is we got synchronous operation inside the iterator, which ended up maxing out the call stack.

If you really cannot avoid mixing synchronous and asynchronous code in an iterator (most of the times you can!), one simple workaround is to wrap synchronous code inside a process.nextTick call, so you clean up the current stack frame and instead of keep increasing the size.


var async = require('async');
var a = [];

for(var i = 0; i < 3040000; i++) {
 a.push(i);
}

async.forEachSeries(a,
 function (item, cb){
  process.nextTick(function () {
   console.log(item);
   cb();
  });
 },
 function () {
  console.log('all done');
 }
);

This issue does not only apply to forEachSeries, but also other function calls like mapSeries, whilist, until, etc. Here is a more detailed discussion thread, where people proposed a patch to add async.unwind to fix the error.

Wednesday, April 25, 2012

JSLint: don't make functions within a loop

Interestingly got this JSLint warning "don't make functions within a loop" today, scratched my head a bit and realized it might be due to the fact that function inside a loop is very prone to cause errors. You are expecting to get different values for the collections you are looping through in each iteration, but it ended up all the same as the value from the last iteration. This post explains it well with a straightforward example.

Another post suggests having functions inside the loop actually cause performance degrade as well, quite interesting experiments.

Mockery: easy mocking in Node.JS

Many JavaScript mocking frameworks are not specifically designed for Node.JS, so when we need to mock built-in Node.JS modules or npm modules, things get quite ugly and hacky.

One approach is to expose __module from module and replace the things you want to test with the mocks. I am not sure if this would work for the built-in modules and it is quite dangerous to mess with __module. Here is the example borrowed from this StackOverflow post:

// underTest.js
var innerLib = require('./path/to/innerLib');

function underTest() {
    return innerLib.toCrazyCrap();
}

module.exports = underTest;
module.exports.__module = module;

// test.js
function test() {
    var underTest = require("underTest");
    underTest.__module.innerLib = {
        toCrazyCrap: function() { return true; }
    };
    assert.ok(underTest());
}


Another approach is to use Dependency Injection (DI). Basically, you pass in mock object through a overwritten module.exports. This post explains well with an example that I borrow below. But do you really want to change the way you invoke require? How about when you are using someone else's library that you cannot even change? What if there are many mock objects you need to pass in?

// overwrite the require with one that accepts a mock object
module.exports = function(http) {
  var http = http | require('http');
  // private functions and variables go here...

  //return the public functions
  return {
    twitterData: function(callback) {
     http.createClient(...etc...);
    }
  };
}

// use it *normally*
var twitter = require('twitter')();

// use it in the test
var mockHttp = { createClient: function() { assert(something); } };
var twitter = require('twitter')(mockHttp);
//do some tests.


So, finally, a more practical solution needs to modify how Node looks up and loads modules. We initially were using some hacks like these:

// Mock native modules, e.g. http
var mockHttp = { request : function () {} };
require.cache['http'] = { exports: mockHttp };

// Mock non-native modules
var path = './test';
var absPath = require.resolve(path);
var mockTest = {...};
require.cache[absPath] = { exports : mockTest };

But this is not always reliable, you need to pay attention to the order you overwrite the modules, be careful with nested require, also it is difficult to reuse the mock objects between different tests, etc.


Finally, a colleague Martin Cooper implemented an elegant solution that makes unit testing with mock objects easy as a breeze. It is called Mockery.

- It supports nested require cases.
- It gives warning about modules that are not mocked out (you can use registerAllowable if you are sure you don't need to mock those out).
- It manages life cycle of mock objects cleanly. You can easily use different mock objects for the same module in different tests (Node only loads a module or mocked module once throughout the process and Mockery can help clean it up).

Here is a quick example how to use it (with YUITest, but you can use Mockery with any testing framework of your choice).

- YUITest can be installed through "npm install yuitest".
- Mockery is installed through "npm install mockery".
- To run the test, do "node node_modules/yuitest/cli.js fsclient.test.js".

///////////////////
//fsclient.js
///////////////////
var fs = require('fs');

function getDate() {
    var today = new Date();
    return today.toUTCString();
}

function getFileContent(filename, callback) {
    fs.readFile(filename, function (err, content) {
        if (err) {
            callback(err);
        } else {
            callback(null, getDate() + "\n" + content);
        }
    });
}
module.exports.getFileContent = getFileContent;
///////////////////
//fsclient.test.js
///////////////////
var YUITest = require('yuitest');
var Assert = YUITest.Assert;
var TestCase = YUITest.TestCase;
var mockery = require('mockery');
var sut = '../fsclient';
var client;

var fsMock = {
 readFile: function (filename, callback) {
        if (filename === 'error') {
            callback(
                new Error('error reading file: ' + filename)
            );
        } else {
            callback(null, 'file content: hello!');
        }
    }
};

var tc = new TestCase({
    'name': 'demo yuitest testcase for fs mocking',

    setUp: function () {
        mockery.enable();
        //replace fs with our fsMock
        mockery.registerMock('fs', fsMock);
        //explicitly telling mockery using the actual fsclient is OK
        //without registerAllowable, you will see WARNING in test output
        mockery.registerAllowable('../fsclient');
    },

    tearDown: function () {
        mockery.deregisterAll();
        mockery.disable();
    },

 testGetFileContentError: function () {
        client = require(sut);
        client.getFileContent('error', function (err, content) {
            Assert.isInstanceOf(Error, err);
            Assert.isTrue(err.message.indexOf('error reading file') !== -1);
        });
    },

    testGetFileContentSuccess: function () {
        client = require(sut);
        client.getFileContent('success', function (err, content) {
            Assert.isNull(err, 'should not get error');
            Assert.areSame((new Date()).toUTCString() + "\nfile content: hello!", content);
        });
    }
});

YUITest.TestRunner.add(tc);
YUITest.TestRunner.run();

There are several other Node.JS mocking frameworks like node-sandboxed-module, injectr, which also worth taking a look.


Martin also has another weapon called Sidedoor, which exposes the private functions that are not exposed to the public. It really helps thoroughly testing the code and improve your code coverage. When your boss tells you code coverage needs to be 90%+ and some error branches are really hard to mock, what do you do?

Use Mockery+Sidedoor!


Other references:

- "Testing private state and mocking dependencies" by Vojta Jina
- "Mockery: hooking require to simplify the use of mocks" discussion thread on Node.JS group
- YUITest, now supporting Node.JS testing as well, also provides yuitest-coverage tool that generates code coverage reports to integrate with Hudson/Jenkins CI environment
- Node.JS module: exports v.s. module.exports

Wednesday, February 15, 2012

Janus with jslint Vim plugin

Nowadays, I write quite a bit of JavaScript and uses JSLint command line tool quite a lot. A co-worker recommends a nice Vim plugin that validates your JavaScript code when you are editing and when you save it.

While I was trying to install it for Janus (a Macvim clone that I uses), I ran into two issues:

1. By default, when you run rake inside the git cloned directory, the plugin gets installed into ~/.vim. For Janus, user customized plugins should go to ~/.janus, and Janus will load them automatically. For details, please check out the Customization section of the Janus documents.

So, to get around:

- create a jslint directory in ~/.janus
- edit vim plugin's Rake file, replace line 38 File.expand_path("~/.vim") with File.expand_path("~/.janus/jslint"), you get the idea.
- run rake from inside vim plugin directory and it should install into the correct Janus directory

2. The second issue is that I keep getting warning when I start trying out the plugin, something like "s:cmd" not defined. Did some poking around and it seems like ftplugin/javascript/jslint.vim is trying to find a JavaScript interpreter (line 62 to 75 for *UNIX systems) and somehow failed.

I am on Snow Leopard, which comes with an acceptable interpreter "jsc" at /System/Library/Frameworks/JavaScriptCore.framework/Resources/jsc, so not really sure why it did not work. I ended up just install Node.JS (for Mac users, highly recommend using Homebrew, just follow instructions here) and added node to my PATH. And it takes care of these errors.

Now, enjoy the jslint plugin and be a good JS developer ;-)


P.S. For people not happy with Crockford's personal styles (some styles don't make sense to me either), you can update options in ~/.jslintrc (see examples on jslint.vim site). Or simply use jshint vim plugin instead. JSHint is a more relaxed and reasonable fork of JSLint.

Sorry for going off the topic, but it is quite funny to read about why the original developer forked JSLint, especially entertaining are the comments. For example, Crockford's response to JSHint:
When asked for his "feelings on JSHint" Crockford replied "There are many stupid people in this world, and now there is a tool for them."

Saturday, January 28, 2012

Lua Development Tools Koneki, IDE based on Eclipse

Just got a comment from developers of Lua IDE Koneki on my previous post about setting up Eclipse-based Lua IDE. So, I gave it a try and it is quite neat, I really hope the developers can keep it moving and make it a great default IDE for Lua developers ;-) As of now, I feel it is not yet conveniently as useful as LuaEclipse I posted about a while back. But the project is under active development and it will become better every day, I am sure.

Here is a quick rundown what I have tried so far:

1. Installation: they offer both standalone and update site, I just installed through update site following their instructions. I am on Eclipse 3.7.1, btw. I also got lua and luarocks installed through Homebrew (by default they are installed in /usr/local/bin).

2. Create a new Lua project: create a new Lua project, add source files under src, similar to the typical Java projects.

3. Ready to run? I feel stuck at this step at first since there is no launch configuration that allows me to configure the local Lua environment. After running through several threads, it seems Koneki does not support launch configuration yet. It only supports "remote debug launch configuration".

But the developer also provides this workaround using "External Tools" and I got it to run and show the results in Console View. This is the sample configuration (you need to select the main.lua before run this external tool configuration). For the meaning of Eclipse variables like ${workspace_loc}, ${resource_loc}, see Eclipse external tools documentation.


4. Debugging. LDT supports remote debugging via DBGP, you can follow the LDT user guide to set it up.

Many thanks to the LDT developers for this nice IDE. I wish the launch configuration can be added so beginners like me can get started with Lua and LDT quickly. Also, I wish the configuration of debugging and remote debugging can be integrated into LDT.

Monday, January 23, 2012

Unbounded function wrappers

I was reading "JavaScript Garden" (highly recommend for JS beginner/intermediate) and could not understand the concept of the "fast, unbound wrappers" for functions. See Function arguments for the example.

function Foo() {}

Foo.prototype.method = function(a, b, c) {
    console.log(this, a, b, c);
};

// Create an unbound version of "method" 
// It takes the parameters: this, arg1, arg2...argN
Foo.method = function() {

    // Result: Foo.prototype.method.call(this, arg1, arg2... argN)
    Function.call.apply(Foo.prototype.method, arguments);
};


Luckily I found this Stackoverflow post that explains the idea behind it. It took me a while to wrap my head around Function.call.apply ;-)

So, the basic idea is that we have a function defined in a class, but we want to use it without binding to specific object, maybe the concept of static method in Java? So, instead of creating an object and invoke the function on the object, we define the function as a property of the class, not on its prototype.