Wiring up dependencies in node.js – wrap up of my best practices

Quite a lot of friend developers that make their first baby steps with node.js keep asking me about the “right” way to share database resources respectively dependencies in general among controllers, business logic or service objects. Now, there’s actually one universal wisdom in the world of node.js: there is no “right” way of doing things (have a look at the controller samples from express.js ).

There are two good reasons why: First, most node.js frameworks including its core are very basic, offering modular and highly simplistic, fundamental but reusable code. They don’t make any assumptions about how you will use them, leaving it up to you to wire everything together as you want. Second, Javascript as a functional language offers many more options to shoot yourself in your foot than class oriented / C-like languages. What’s missing (and maybe even unwanted) in the node.js universe is a sophisticated framework like Spring, Rails or Symfony that offers best practices under the hood you could rely on.

In this article I’d like to tell my story of how I first tried to transfer my knowledge from these “sophisticated” frameworks (usually IoC containers) to express.js just to recognize the far simpler, nodejs-“native” approach to achieve the same results. The first three examples I’m presenting are just evolutions of ideas I had until I noticed that in node.js you can take much simpler ways – so be warned that some of the upcoming code might seem unnecessarily blown up: the resolution you might want to comment on can be found at the very end.

The old world

Connecting to some kind of server-side database layer is a pretty straight forward task in your favorite non-JS language. Have a look at some pseudo-code:

Connection con = Driver.connect("some://driver.specific@connection.string:for/your/database");
ResultSet result = con.query("SELECT foo FROM bar WHERE id=1337");
Entities entities = new ArrayList;
foreach(result as r) { 
    Entity e = Driver.hydrate(r); 
    entities.add(e);
}   

If you’re a real node.js greenhorn, let me quickly explain why this kind of code will never work on node. Since Javascript (really!) executes in one thread at a time lengthy operations like connecting or querying a database must not block the main execution loop – otherwise the application won’t be able to respond other incoming requests. Instead of waiting for the database to return a connection (like e.g. Java does), V8 will immediately execute the subsequent code. Regarding the example above con might not have been initialized yet when `con.query is executed. In Javascript you handle this kind of asynchronous events using callbacks, so in node.js the above example could be pseudo-coded as:

var entities = [];

driver.connect(connectionString, function(err, con) {
    var hydrator = new driver.Hydrator;
    con.query("SELECT ...", function(err, result) {
        _.each (result, function(r) {
            hydrator.hydrate(r, function(err, entity) {
                entities.push(entity);

                //business logic here

            });
        });
    });
});

You immediately notice the main problem in functional code: some call it the “Javascript pyramid of death”. It’s built on subsequently registered callbacks. I won’t dig deep into solutions to that issue here (promises are currently the best solution and they’re widely adapted ) but I want you to have a look at the first line that connects to the database and serves as the root of our pyramid. On platforms like PHP or Java you would have a single place where you connect to your database. Then you would either wire that connection to clients that want to use it or ask some container to hand it over back to you (and initialize it if it wasn’t before). Let’s have a look at that pattern in a container managed environment (far from being exactly IoC, but you should get the idea):

“Java” pseudo-code:

class Container {
    resources = array();  
    con = null;

    public function getConnection() {
        if (null == this.con) {
            this.con = Driver.connect(...);
        }
        return this.con;
    }
}  

class FooController {

    @Inject("container")
    private container;

    db = this.container.getConnection().query("...");
}

The idea behind this pattern is: there’s some godlike mega-registry (the IoC-Container) knowing, configuring and instantiating all your dependencies. If you need something you either annotate your dependencies and let Mr Registry inject it at startup time or call Mr Registry and ask for a fully configured and initialized resource (service, bean, you name it).

IoC-like coding in express.js / node.js

Adapting this pattern in node.js leads to rather uncomfortable code. Lets start with an app.js to illustrate that (Again: beware that I wouldn’t recommend to use this kind code but you could definitely do so).

/code

app.js

var express = require("express");
var http = require('http');
var Controller = require('./controller.js');
var Container = require("./ioc.js");

var app = express();
app.use(express.bodyParser());

app.set('port', process.env.PORT || 3000);

var container = new Container();
new Controller(container).route(app);

app.use(app.router);

http.createServer(app).listen(app.get('port'), function(){
    console.log('Express listens on port ' + app.get('port'));
});

ioc.js

var sqlite3 = require("sqlite3");
var Container = module.exports = function() {
    var db = null;
}

Container.prototype = {
    initializeDb: function(db) {
        db.run("CREATE TABLE testing " +
            "(id INTEGER PRIMARY KEY AUTOINCREMENT, " +
             "info TEXT)", function(err) {
            console.dir(err);
        });
    },
    getDb: function() {
        if (null == this.db) {
            this.db = new sqlite3.Database(":memory:");
            this.initializeDb(this.db);
        }
        return this.db;
    }
}

controller.js

var Controller = module.exports = function(container) {
    this.container = container;
}

Controller.prototype =  {
    indexAction: function(req, res) {
        var db = this.container.getDb();
        db.all(
           "SELECT * FROM testing",
            function(err, rows) {
                res.json(rows);
            });
    },
    fooAction: function(req, res) {
        var db = this.container.getDb();
        db.get(
            "SELECT * FROM testing WHERE id = $id",
            {$id:req.params.id},
            function(err, row) {
                res.json(row);
            });
    },
    addAction: function(req, res) {
        var db = this.container.getDb();
        db.run(
            "INSERT INTO testing (info) VALUES ($info)",
            {$info: req.body.info},
            function(err) { //this: statement
                res.json({ lastId: this.lastId});
            });
    },
    route: function(app) {
        app.get('/foo', this.indexAction.bind(this));
        app.get('/foo/:id', this.fooAction.bind(this));
        app.post('/foo', this.addAction.bind(this));
    }
}

If you run this example and access GET /foo it’ll be called back with the error message “Error: SQLITE_ERROR: no such table: testing“. Notice that initializeDb has been called by getDb but while it tried to execute the initial CREATE statement, the single thread already went on and executed the index action. Since the initializeDb callback has not been handled yet (it’s going to be handled right after the index action has finished) the SELECT statement cannot find the table yet.

To remedy that situation, we could use callbacks in our client code, like this:

fooAction: function(req, res) {
    this.container.getDb( function(db) {
        db.get("SELECT ...", {}, function() { ... });
    });

That way we’re littering the container’s getDb interface with a callback parameter. Instead, people choose to use the so called promise pattern at this point. There are some libraries out there that get the job done; Q is one of the most powerful of them and besides many other features it offers a deferred interface that deals with that problem.

IoC-like code with promises

/code

container.js

Container.prototype = {

    initializeDb: function(db, callback) {
        db.run("CREATE TABLE testing " +
            "(id INTEGER PRIMARY KEY AUTOINCREMENT, " +
            "info TEXT)", callback);
    },
    getDb: function( ) {
        var deferred = Q.defer();
        if (null == this.db) {
            var self = this;
            this.db = new sqlite3.Database(":memory:");
            this.initializeDb(this.db, function(err) {
                if (err) {
                    deferred.reject(new Error(err));
                } else {
                    deferred.resolve(self.db);
                }
            });
        } else {
            deferred.resolve(this.db);
        }
        return deferred.promise;

    }
}

controller.js

    ...
    indexAction: function(req, res) {
        this.container.getDb().then( function(db) {
            db.all("SELECT * FROM testing",
                function(err, rows) {
                    res.json(rows);
                });
        });
    }
    ...

Starting the app after container setup has finished

That looks only little better but definitely not really usable yet. So lets take a last attempt to fix things up. Let’s tell the container to initialize everything and start the application once the basic initialization has finished.

/code

app.js

var container = new Container();
container.initialize().then( function() {
    new Controller(container).route(app);
    app.use(app.router);

    http.createServer(app).listen(app.get('port'), function(){
        console.log('Express listens on port ' + app.get('port'));
    });
});

container.js

Container.prototype = {

    initializeDb: function(db, callback) {
        db.run("CREATE TABLE testing " +
            "(id INTEGER PRIMARY KEY AUTOINCREMENT, " +
            "info TEXT)", callback);
    },
    getDb: function( ) {

        if (null == this.db) {
            var deferred = Q.defer();
            var self = this;
            this.db = new sqlite3.Database(":memory:");
            this.initializeDb(this.db, function(err) {
                if (err) {
                    deferred.reject(new Error(err));
                } else {
                    deferred.resolve(self.db);
                }
            });
            return deferred.promise;
        } else {
            return this.db;
        }

    },
    initialize: function() {
        var dfd = Q.defer();
        this.getDb().then( function() {
            dfd.resolve();
        });
        return dfd.promise;
    }
}

controller.js

var Controller = module.exports = function(container) {
    this.container = container;
    this.db = container.getDb();
}

Controller.prototype =  {
    indexAction: function(req, res) {
        this.db.all("SELECT * FROM testing",
            function(err, rows) {
                res.json(rows);
            });
    },
    ...
}

This looks like a good start for configuring action controllers with container based dependencies on a very low level. This concept can be extended to have a configurable container, to load a dependency tree and prepare singleton services, even to lazy load services using proxy objects. Et voila: welcome back to the good old Spring / Symfony world. You can write your code that way, it’s going to work pretty much as expected (I worked that way for quite some time)

Out of the rabbit hole

Wake up, Alice – you’re not in Wonderland anymore. Here’s what I’m doing in node.js these days; pretty, lean and simple.

/code

DB.js

var sqlite3 = require("sqlite3");
var db = new sqlite3.Database(":memory:");

db.run("CREATE TABLE testing " +
    "(id INTEGER PRIMARY KEY AUTOINCREMENT, " +
    "info TEXT)");

module.exports = db;

app.js

var express = require("express");
var http = require('http');
var Controller  = require('./controller.js');
var DB = require("./DB.js");

var app = express();
app.use(express.bodyParser());

app.set('port', process.env.PORT || 3000);

app.use(app.router);

app.get('/foo', Controller.indexAction.bind(Controller));
app.get('/foo/:id', Controller.fooAction.bind(Controller));
app.post('/foo', Controller.addAction.bind(Controller));

http.createServer(app).listen(app.get('port'), function(){
    console.log('Express listens on port ' + app.get('port'));
});

controller.js

var DB = require("./DB.js");

module.exports = {
    indexAction: function(req, res) {
        DB.all("SELECT * FROM testing",
            function(err, rows) {
                res.json(rows);
            });
    },
    fooAction: function(req, res) {
        DB.get(
            "SELECT * FROM testing WHERE id = $id",
            {$id:req.params.id},
            function(err, row) {
                res.json(row);
            });
    },
    addAction: function(req, res) {
        DB.run(
            "INSERT INTO testing (info) VALUES ($info)",
            {$info: req.body.info},
            function(err) { //this: statement
                res.json({ lastId: this.lastId});
            });
    }
}

The “magic” that might seem unusual to mature developers coming from class oriented environments lies in the “module” concept that’s one of node’s cornerstones. A module is the encapsulation of state and behavior, it can be used in a service-like fashion like I did in the last example. Notice, that I’m requiring DB.js in app.js without even using it. That way node.js executes the code inside once and keeps the reference in module.exports – the database therefore is prepared when a controller is using it. Well, not exactly: if the preparation / setup of resources takes really long, the application is up before initialization has finished (try wrapping the database in a timeout, I left the code as comment in the repo). But what’s more important: taking this approach you don’t have to take much care about your dependencies but rather can access them from any module you want simply by requiring them. The db variable in DB.js always refers to the very same instance.

tl/dr Lessons learnt: node.js and expressjs don’t propose a structure for managing dependencies and wiring them up with your clients. While you could write your code in a rather classical way, it’s mostly much simpler to use node’s builtin concepts.

PS. I just found this stackoverflow question that’s underlining the concept I tried to explain here in short. Good, that I’m not alone with my opinion 😉

Advertisements

Apache resigns from JCP: The end of free Java?

I followed the news about Oracle’s Java acquisition pretty intense in the last weeks and today the summit seems to be taken: The Apache foundation left the Java community process that accounts for specifications found in nearly all J2SE + J2EE components. I think this is a destastrous step for the overall development and progress of the Java ecosphere. Apache is a foundation that embraces open source more than no one else. Many products are built upon Apache technology, e.g. the popular Apache Server (that has nothing to do with Java), Tomcat, Geronimo, ANT, not to mention all the little helper projects and frameworks (JSF) that help making web development easier, robust and reliable.

With the retirement of Apache lots of knowledge is draining from the JCP and the foundation itself hasn’t told yet what they’re about to do next. Maybe they start off working  on their open Java implementation Harmony but I doubt that it’ll gain lots of popularity as long as the “official” Java is maintained by a commercial and therefore “believable” company like Oracle. From my point of view Apache’s decision will lead to a slow but remarkable decrease of Java usage in the development industry. There are plenty of companies, organizations and OS projects that will suffer from it since the foreseeable forking of Java technology will lead to a fragmentation that’s going to kill Java’s fundamental strive to unify operating systems and runtimes untilizing one open binary compatible platform. Shortly put: Java is nothing more than a formally open .NET framework, controlled by a commercial company.

This is not the way open software should be developed. One could have said that when Google launched  the binary incompatible Dalvik VM on Android over that the search giant is going to trial with Oracle. But at least Google tries to implement an open source process behind the popular mobile operating system.

Since fragmentation of technology will lead to a mess of implementations Java technology is about to loose it’s spearhead position in the server environments in the middle future. I’d estimate that by 2015 many projects won’t be launched on Java anymore but choose a meta oriented approach like Ruby, PHP or Python. It’s time for every developer to think about her future. Do you want to develop against a fragmented and unvisionary, monolithic controlled architecture like Java or .NET or do you want to concentrate on upcoming standards like Canvas/HTML5, PHP (APC), Python (Django), Ruby ? It’s your decision. I’ve made mine.

Official announcement of Apache quitting the JCP

Thanks to Spring Community: REST-Client for Android

You know what really pissed me off when writing code for Android? There literally is no genuine REST client in that system, even though the original Android API uses REST mechanisms to store and retrieve data in their ContentProvider architecture. Funny enough that the only community project that I know of trying to remedy that issue was a pain in the ass to install (because of Maven dependencies) and in the end nearly unusable in terms of architecture.

Today the Spring Community published a first milestone for Android addons and you guessed it: it comes with a fully fledged REST client (so says the announcement, I haven’t tested yet). The good thing about it: if you’re used to Spring’s ORM template classes (we use iBATIS) you’ll quickly recognize the call architecture for the Android REST client is quite similar. Here’s an example call:

RestTemplate restTemplate = new RestTemplate();
restTemplate.setRequestFactory(new CommonsClientHttpRequestFactory());
String url = "https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q={query}";
String result = restTemplate.getForObject(url, String.class, "Thanksgiving");

aint that cool stuff? Whilst other authors try to use a common library like Apache HttpClient, Spring allows you have connections created using a RequestFactory. So if you want to use a more reliable another Http layer feel free to implement one. The template mechanism for REST-like URLs also feels comfortable so the last thing to say: incredible but far too late, guys!

Oh, BTW. The only other feature that Spring for Android currently is offering is a streamlined logging client. Also very important but IMHO not made for the catwalk. Pls have look by yourself.

REST API Reference

Spring Android Product Page

Read of the week: Exception Handling AntiPatterns

That’s a very nice article about exception handling and how it’s not done.

http://today.java.net/article/2006/04/04/exception-handling-antipatterns

unfortunately I see so much code floating around that makes use of that anti-pattern that I strongly urge every developer reading this to take note of that article (and at least quickly read over it). It’s also a good read for C++ devs that still have the opinion that exceptions are bad by design – at last they aren’t if you use them correctly.

Happy throwing