Click here to Skip to main content
Click here to Skip to main content

Node.Js And Stuff

, 11 Feb 2013 CPOL
Rate this:
Please Sign up or sign in to vote.
Small demo app using Node.Js/Socket.IO/MongoDB/D3.Js and jQuery.

Contents

Introduction

It has been a while since I wrote an article here at CodeProject, so I apologize for that. I have been pretty busy doing, um nothing I guess.

Anyway late last year I found a bit of spare time, so I looked at my to-do / must learn list, and sitting there glaring at me was "Node.Js" in 60 ft letters. So I thought OK, why not give Node.js a try. At the time I was doing a fair amount of ASP MVC / jQuery at work, so thought now would be a good time to do this, as I kinda already had my head in the web zone at work, so it would be a good time to give Node.js a go.

So this article is me playing around with Node.js a bit, and coming up with a small demo project. The demo project is tiny as I did not want to overload people with too much, that said the choice of the demo app I have made ensures (at least in my mind it does) that by the time you have read this article you will get to learn about the following elements:

  • Node.Js itself and how a Node.js application is structured (well, one potential way it could be structured, that is)
  • Express which is a MVC type Node.js framework
  • NPM which is Node Package Management
  • Jade view engine
  • Stylus which is a CSS generator that works a lot like the popular SASS/LESS CSS generators
  • Socket.IO which is a Web Socket package for Node.JS (star of the show if you ask me)
  • D3.Js which is a data visualization JavaScript library
  • Mongo DB document database Node.js integration

Just for the record, Node.js does not need to be used to create web sites. It can be used just to create servers, it just so happens I have chosen to create a web site with it. 

Brief Node.Js Overview

As I have stated I have decided to write this article using Node.js, so what exactly is Node.js? Let's see a few descriptions of it before we carry on, shall we?

Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.

-- http://nodejs.org/ 

Node.js is a server side software system designed for writing scalable Internet applications, notably web servers. Programs are written on the server side in JavaScript, using event-driven, asynchronous I/O to minimize overhead and maximize scalability.

Node.js is a packaged compilation of Google's V8 JavaScript engine, the libUV platform abstraction layer, and a core library, which is itself primarily written in JavaScript.

Node.js was created by Ryan Dahl starting in 2009, and its growth is sponsored by Joyent, his employer. Dahl's original goal was to create the ability to make web sites with push capabilities as seen in web applications like Gmail. After trying solutions in several other programming languages he chose JavaScript because of the lack of an existing I/O API. This allowed him to define a convention of non-blocking, event-driven I/O.

--http://en.wikipedia.org/wiki/Nodejs

Here is a trivial node example, where this is a complete implementation of Hello World as an HTTP server in Node.js:

var http = require('http');
 
http.createServer(
  function (request, response)
  {
    response.writeHead(200, {'Content-Type': 'text/plain'});
    response.end('Hello World\n');
  }
).listen(8000);
 
console.log('Server running at http://localhost:8000/');

JS Everywhere

One of the big things that comes by using Node.Js is that you are effectively using JavaScript on both the client and server side code bases, so if you are familiar with JavaScript the promise is that this single language on both sides of the wire could speed up your development time quite substantially.

Prerequisites

Before we start the break down of how the demo app works, if you intend to actually run the demo code you will need to grab a few things before you start. These are listed below:

1. Mongo DB

You will need to download and unzip Mongo DB, which is available from here (I chose the Windows 64-bit version, but you will need to pick one that suits your machine specifics):

2. Node.js

You will need to download Node.js, which is available from here:

Node.Js Basics

In this section we will try and examine the inner workings of the demo app. By the end I would hope you know a bit about Node.js and some of its standard components.

Required Packages

Node.Js is geared around packages, kind of like NuGet, where you request a certain package from the package manager and it will download and place the relevant package files in the correct location.

Package management in Node.Js is achieved using the "Node.js Command Prompt", where we use NPM.exe (Node Package Manager) to install the packages.

Here is an example (in fact this is the full example of how I set up the demo codes packages).

These are the steps I followed to install node packages (You will not have to do this)

It should be noted that the following steps also create a dummy Express (more on this later) application shell.

  1. From node command prompt
  2. mkdir WebSocketDemoApp
  3. cd WebSocketDemoApp
  4. express -c stylus
  5. npm install -d
  6. npm install socket.io 
  7. Edit package.json to include Mongo DB ("mongo DB": ">= 0.9.6-7")
  8. npm install -d

When you install a package using NPM.exe you should see that the packages file "package.json" is updated (you can also tweak this file a bit if you have very specific requirements, as I did where I wanted to ensure the Mongo DB driver must be > 0.9.6-7 version).

Package.Json

Here is the resulting packages file "package.json" for the attached demo app:

{
  "name": "application-name" ,
  "version": "0.0.1" ,
  "private": true ,
  "scripts": {
    "start": "node app"
  } ,
  "dependencies": {
    "express": "3.0.0rc4" ,
    "jade": "*" ,
    "stylus": "*" ,
	"Mongo dB": ">= 0.9.6-7"
  }
}

What The Packages Typically Look Like

When Node.Js / NPM.exe have successfully downloaded and installed packages they will be placed in a "node_modules" folder:

If we examine one of the package folders, say "Express", we can see that all the relevant files are present:

Note: For the demo app I have obviously installed all the relevant packages for you, so you do not need to do any of that.

So that is essentially how you get packages installed, we will be talking more about packages as the demo app makes use of several packages, so don't worry you will see more on how they are used.

Express (Node.js Package)

Express is a standard Node.js package that provides standard features such as:

  • Routing
  • Routing data
  • Rendering
  • HTTP Listener

When you create a new Express application, you also get a certain folder structure and a skeleton application file. Shown below is a typical empty Express project:

We will dissect this a bit further as it will help us when we get to look at the real demo app code:

App skeleton

The actual "Node.js application" you get when you create a new Express application is called "pap's" which contains some skeleton code for you, that provides the following:

  • Configuration
  • Routing
  • Http Listening/Server

Here is what that "app.js" starts out like before it is changed, as it has been for this article's demo code:

/**
 * Module dependencies.
 */

var express = require('express')
  , routes = require('./routes')
  , user = require('./routes/user')
  , http = require('http')
  , path = require('path');

var app = express();

app.configure(function(){
  app.set('port', process.env.PORT || 3000);
  app.set('views', __dirname + '/views');
  app.set('view engine', 'jade');
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
  app.use(require('stylus').middleware(__dirname + '/public'));
  app.use(express.static(path.join(__dirname, 'public')));
});

app.configure('development', function(){
  app.use(express.errorHandler());
});

app.get('/', routes.index);
app.get('/users', user.list);

http.createServer(app).listen(app.get('port'), function(){
  console.log("Express server listening on port " + app.get('port'));
});

It can be seen that this is mainly broken down into three parts:

  1. App configuration
  2. Setting up the routing
  3. Creating a Http server and starting to listen

Public folder

This contains things that are, um well, public, such as Images/JavaScript/CSS, where you would put the files into the relevant folder:

Routes folder

This folder contains all the code which manage the routes and their views. So you would typically have

  • GET/POST handlers
  • Rendering views
  • Building up models

Here is a small example of one of the views you get with a fresh Express application:

/*
 * GET home page.
 */

exports.index = function(req, res){
  res.render('index', { title: 'Express' });
};

Views folder

This folder holds the views for your application. Since Jade is the default view engine you get with Express, all the views you get with a fresh Express application will be Jade view files. As we will be covering Jade in just a moment, we will save that discussion for the next section. 

Jade

Jade is one of the many view engines you can use with Express. It also happens to be the default one that you get when you choose to use the Node.Js Express package.

Jade is very sophisticated in loads of areas, and really dumb in one. Unfortunately that one area where it is dumb turns out to be a real sticking point for me.

The Pros

This is what the Jade site lists as the features:

  • client-side support great readability
  • flexible indentation
  • block-expansion
  • mixins
  • static includes
  • attribute interpolation
  • code is escaped by default for security
  • contextual error reporting at compile & run time
  • executable for compiling jade templates via the command line
  • html 5 mode (the default doctype)
  • optional memory caching
  • combine dynamic and static tag classes
  • parse tree manipulation via filters
  • template inheritance
  • block append / prepend
  • supports Express JS out of the box
  • transparent iteration over objects, arrays, and even non-enumerables via each
  • block comments
  • no tag prefix
  • filters

I guess it may be better to see an example. So let's consider one of the two demo app views, where we can examine the Jade markup and the HTML it produces.

Here is my "layout.jade" page (think master page if you are coming from Microsoft land like I do):

doctype 5
html
  head
  link(rel='stylesheet', href='http://www.codeproject.com/stylesheets/style.css')
  script(src='http://www.codeproject.com/javascripts/jquery-1.8.2.min.js')
  script(src='http://www.codeproject.com/javascripts/d3.v2.js')
  link(rel='stylesheet', href='http://www.codeproject.com/jquery-ui-1.9.1.custom/css/ui-lightness/jquery-ui-1.9.1.custom.css')
  script(src='http://www.codeproject.com/jquery-ui-1.9.1.custom/js/jquery-ui-1.9.1.custom.js')
  block head
body
  div(id='mainContent')
    img(src='http://www.codeproject.com/images/Header.png')
    block content

And here is the demo apps view "home.jade" which is an extending view (that's Jade's way of dealing with Master page like functionality).

Pretty precise, huh?

extends layout

block head
  link(rel='stylesheet', href='http://www.codeproject.com/stylesheets/home.css')
  script(src='http://www.codeproject.com/javascripts/home.js')

block content
  form(method='post', id='homeForm', action='http://localhost:2000/home')
    div(id='dialog', title='error', style='display:none;')
      p(id='homeWarn') You need to supply a valid email
    div(id='NewDetailsArea')
      p Enter your email address, and then click enter 
      | <input type='text' id='email' name='email' class='email'></input>
      div#homeSubmit
        input(type='submit', value='Enter', id='enterEmail')

And here is what is produced:

<!DOCTYPE html>
<html>
<head>
</head>
<link rel="stylesheet" href="http://www.codeproject.com/stylesheets/style.css">
<script src="http://www.codeproject.com/javascripts/jquery-1.8.2.min.js"></script>
<script src="http://www.codeproject.com/javascripts/d3.v2.js"></script>
<link rel="stylesheet" href="http://www.codeproject.com/jquery-ui-1.9.1.custom/css/ui-lightness/jquery-ui-1.9.1.custom.css">
<script src="http://www.codeproject.com/jquery-ui-1.9.1.custom/js/jquery-ui-1.9.1.custom.js"></script>
<link rel="stylesheet" href="http://www.codeproject.com/stylesheets/home.css">
<script src="http://www.codeproject.com/javascripts/home.js"></script>
</html>
<body>
    <div id="mainContent">
        <img src="http://www.codeproject.com/images/Header.png"><form method="post" id="homeForm" action="http://localhost:2000/home">
        <div id="dialog" title="error" style="display: none;">
            <p id="homeWarn">
                You need to supply a valid email</p>
        </div>
        <div id="NewDetailsArea">
            <p>
                Enter your email address, and then click enter
            </p>
            <input type='text' id='email' name='email' class='email'></input><div id="homeSubmit">
                <input type="submit" value="Enter" id="enterEmail"></div>
        </div>
        </form>
    </div>
</body>

I can't really go into every aspect of Jade, if you want to know more, you can read about it here: https://github.com/visionmedia/jade#readme.

The 1 Big Con (At least in my opinion)

Jade uses semantic white space, and it is very very fussy about mixing tabs/spaces. Even when I was a gazillion % sure I had used all spaces, it would complain. And when it complains, all you get rendered instead of the view is a crappy Exception message.

Now it could be more you use it the better you get with it. I just found I was forever getting these Exceptions, and was at a real loss to fix them at times. It felt like they got fixed more by luck than anything else.

Being fair to Jade this is the only complaint I have, it really is a very capable view engine.

Stylus

Now I don't profess to like CSS very much, but being a software developer, I am interested in all aspects of software design, so I have taken note of things like SASS/LESS for greatly improving how you work with CSS. Where these frameworks provide things like

  • Variables within CSS
  • Nested CSS declarations (more explicit)
  • Mix-ins
  • Scoping

As it turns out there is a very funky package you can use with Node.js that you can choose to use with the Express package, which is what we get when we use "NPM.exe" and issue the command "npm express -c stylus" which we talked about earlier. As you may have guessed from that command line, it is called "Stylus" and it does most of the things listed above and more.

Here is an example of one of the demo apps "Stylus" files, and the resultant CSS files. I am not using all the tricks, but rest assured "Stylus" is very capable, just check learnboost.github.com/stylus/ for more details

.Styl file

Here is the raw .Styl file, notice how I am using global variables there, and that there are no semi-colons at all:

foreground-color = white
background-color = black
font =  12px Verdana
general-margin = 10px

body
  padding: 50px
  font: font 
  font-weight:normal
  background-color: background-color
  width: 100%
  height: 100%
  margin: none
  padding: none

a
  color: #00B7FF

.popupText
 color : background-color

#summary
	 text-align: center

p 
 color : foreground-color

input[type="submit"], input[type="button"] 
  border: 1px solid black
  padding: 4px
  width: 120px
  margin-top: general-margin 
  margin-right: general-margin 
  height:23px
  font: font 
  foreground : foreground-color

input[type="text"] 
  height:23px
  font-weight:normal

.nodetext
  pointer-events: none
  font: font
  fill : foreground-color

.circle 
 z-index:999

.link
  stroke: foreground-color

#graph
  width: 1000px
  height: 400px
  margin-left: auto
  margin-right: auto
  border: 1px dashed foreground-color

#mainContent
  width: 1000px
  height: 600px
  margin-left: auto
  margin-right: auto

#NewDetailsArea
  margin-left: 15px
  margin-top: general-margin

#email, #NewPersonEmail, #sourcePeople, #targetPeople 
    width : 350px
    margin-left: 0px
    margin-top: general-margin
    margin-bottom: general-margin
    margin-right: general-margin

Resultant .css file

And here is the resulting CSS file:

body {
  padding: 50px<span class="code-none">;
  font<span class="code-none">: 12px Verdana<span class="code-none">;
  font-weight<span class="code-none">: normal<span class="code-none">;
  background-color<span class="code-none">: #000<span class="code-none">;
  width<span class="code-none">: 100%<span class="code-none">;
  height<span class="code-none">: 100%<span class="code-none">;
  margin<span class="code-none">: none<span class="code-none">;
  padding<span class="code-none">: none<span class="code-none">;
<span class="code-none">}
a <span class="code-none">{
  color<span class="code-none">: #00b7ff<span class="code-none">;
<span class="code-none">}
.popupText <span class="code-none">{
  color<span class="code-none">: #000<span class="code-none">;
<span class="code-none">}
#summary <span class="code-none">{
  text-align<span class="code-none">: center<span class="code-none">;
<span class="code-none">}
p <span class="code-none">{
  color<span class="code-none">: #fff<span class="code-none">;
<span class="code-none">}
input[type="submit"],
input[type="button"] <span class="code-none">{
  border<span class="code-none">: 1px solid #000<span class="code-none">;
  padding<span class="code-none">: 4px<span class="code-none">;
  width<span class="code-none">: 120px<span class="code-none">;
  margin-top<span class="code-none">: 10px<span class="code-none">;
  margin-right<span class="code-none">: 10px<span class="code-none">;
  height<span class="code-none">: 23px<span class="code-none">;
  font<span class="code-none">: 12px Verdana<span class="code-none">;
  foreground<span class="code-none">: #fff<span class="code-none">;
<span class="code-none">}
input[type="text"] <span class="code-none">{
  height<span class="code-none">: 23px<span class="code-none">;
  font-weight<span class="code-none">: normal<span class="code-none">;
<span class="code-none">}
.nodetext <span class="code-none">{
  pointer-events<span class="code-none">: none<span class="code-none">;
  font<span class="code-none">: 12px Verdana<span class="code-none">;
  fill<span class="code-none">: #fff<span class="code-none">;
<span class="code-none">}
.circle <span class="code-none">{
  z-index<span class="code-none">: 999<span class="code-none">;
<span class="code-none">}
.link <span class="code-none">{
  stroke<span class="code-none">: #fff<span class="code-none">;
<span class="code-none">}
#graph <span class="code-none">{
  width<span class="code-none">: 1000px<span class="code-none">;
  height<span class="code-none">: 400px<span class="code-none">;
  margin-left<span class="code-none">: auto<span class="code-none">;
  margin-right<span class="code-none">: auto<span class="code-none">;
  border<span class="code-none">: 1px dashed #fff<span class="code-none">;
<span class="code-none">}
#mainContent <span class="code-none">{
  width<span class="code-none">: 1000px<span class="code-none">;
  height<span class="code-none">: 600px<span class="code-none">;
  margin-left<span class="code-none">: auto<span class="code-none">;
  margin-right<span class="code-none">: auto<span class="code-none">;
<span class="code-none">}
#NewDetailsArea <span class="code-none">{
  margin-left<span class="code-none">: 15px<span class="code-none">;
  margin-top<span class="code-none">: 10px<span class="code-none">;
<span class="code-none">}
#email,
#NewPersonEmail,
#sourcePeople,
#targetPeople <span class="code-none">{
  width<span class="code-none">: 350px<span class="code-none">;
  margin-left<span class="code-none">: 0px<span class="code-none">;
  margin-top<span class="code-none">: 10px<span class="code-none">;
  margin-bottom<span class="code-none">: 10px<span class="code-none">;
  margin-right<span class="code-none">: 10px<span class="code-none">;
<span class="code-none">}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>

It can be seen that the global variables have been substituted where needed, and all semi-colors, braces, etc., have been rendered. What happens when you run a Node.Js/Express app that has been configured to use "Stylus" is that the "Stylus" files *.styl are transformed into standard *.css files.

The Demo App

This section talks about what the demo app does, and also talks you through how to get the demo app up and running on your own box.

What Does The Demo App Do?

In theory what the demo app tries to do is dead simple, and can be broken down into the following steps:

  1. Hosts a Socket.IO WebSocket server
  2. Provide two routes (valid known URLs that will serve content) that a connected client can use
    • /home :
      • Which has a GET, which allows the home page to be viewed. The 1st time this route is called some initial seed data will be placed into the application MongoDB database (Providing there is not already some data present)
      • Which also has a POST which will accept the entered Email address for a new user, and will save that in the application MongoDB database, and the redirect the user to the /d3Demo route
    • /d3Demo
      • Which has a GET which simply grabs all the previously stored Person and Link objects from the application MongoDB database and shows this in a D3.js Force type (think spring connected) graph layout
      • Has a button to add a new Person, which will launch a jQuery UI dialog to allow the new details to be entered. If entered correctly, shall call the Socket.IO "addNewPersonMessage" message to ask the node application server to add a new Person to the application MongoDB database
      • Has a button to add a new Link, which will launch a jQuery UI dialog to allow the new details to be entered. If entered correctly shall call the Socket.IO "addNewLinkMessage" message to ask the node application server to add a new Person to the application MongoDB database
      • Initialises a Socket.IO WebSocket client connection back to the applications Socket.IO WebSocket server, which allows the following messages and callbacks to be sent and received
        • addNewPersonMessage: Message from client to server to add new Person object to application MongoDB database
        • addNewLinkMessage: Message from client to server to add new Link object to application MongoDB database
        • newPersonCallback: Message from server to all clients, which allows client(s) to respond to the new Person that has been added. In this demo this simply means adding the new Person to the D3.js Force type (think spring connected) graph layout
        • newLinkCallback: Message from server to all clients, which allows client(s) to respond to the new Link that has been added. In this demo this simply means adding the new Link to the D3.js Force type (think spring connected) graph layout

If that is all to hard to understand, this diagram may help to illustrate this further (click the image for a larger version):

Running The Demo App

  1. Make sure you have started the "Mongod.exe" process from wherever you unzipped the MongoDB download to (for me "mongodb-win32-x86_64-2.2.1\bin\mongod.exe" is the location for the EXE). If you want to start with a clean database to see how things work, the MongoDB files are available in the "C:\data\db" directory, where the demo app will create a database (providing you don't change anything) called "node-mongo-socketDemo.XXXX", so just delete all those file with that file name
  2. Using the "Node.Js command prompt" from where Node.Js installed for you (usually gives you a menu item for this)  navigate to the folder where you downloaded
    this articles code to, and issue the following command to the Node.js command line : "node app"

If you have successfully started the demo app you should see something like this:

Demo App Structure

The demo app is made up of several folders/parts which can be seen in the screen shot below:

I will endeavor to go through these parts within the rest of this article.

The Demo App Itself

As I have stated the attached demo app makes use of Express which we went through in the Node.Js sections above, it should hopefully be a little bit familiar.

So here is the entire demo app ("app.js") code, which I will describe in a bit of detail:

var express = require('express');
var home = require('./routes/home');
var d3demo = require('./routes/d3demo');
var PersonProvider = require('./public/javascripts/personProvider').PersonProvider;
var personProvider = new PersonProvider('localhost', 27017);
var LinkProvider = require('./public/javascripts/linkProvider').LinkProvider;
var linkProvider = new LinkProvider('localhost', 27017);
var CommonHelper = require('./public/javascripts/commonHelper').CommonHelper;
var commonHelper = new CommonHelper();

var http = require('http');
var path = require('path');
var app = express();

//=============================================================================
//    EXPRESS SETUP
//=============================================================================
app.configure(function(){
  app.set('port', process.env.PORT || 2000);
  app.set('views', __dirname + '/views');
  app.set('view engine', 'jade');
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
  app.use(require('stylus').middleware(__dirname + '/public'));
  app.use(express.static(path.join(__dirname, 'public')));
});

app.configure('development', function () {
    app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
});

app.configure('production', function () {
    app.use(express.errorHandler());
});


//=============================================================================
//    ROUTING
//=============================================================================
app.get('/home', function (req, res) {
    home.homeGet(req, res, commonHelper, personProvider, linkProvider);
});

app.post('/home', function (req, res) {
    home.homePost(req, res, personProvider);
});

app.get('/d3demo', function (req, res) {
    d3demo.d3demoGet(req, res, commonHelper, personProvider, linkProvider);
});


var server = http.createServer(app);
var io = require('socket.io').listen(server);

server.listen(app.get('port'), function(){
    console.log("Express server listening on port " + app.get('port'));
});


//=============================================================================
//    SOCKETS CREATION
//=============================================================================
io.sockets.on('connection', function (socket) {

    socket.on('addNewPersonMessage', function (data) {
        console.log("addNewPersonMessage recieved person (email:'" + data.email + "')");

        personProvider.save({
            email: data.email
        }, function (error, docs) {
            if(error == null) {
                console.log('Sucessfully saved new person');
                console.log(docs[0]);
                io.sockets.emit('newPersonCallback', { _id: docs[0]._id, email: docs[0].email });
            } 
        });
    });


    socket.on('addNewLinkMessage', function (data) {
        console.log("addNewLinkMessage recieved link (source:'" + data.source + "', target:'" + data.target + "')");
        linkProvider.save({
            source: data.source,
            target: data.target
         }, function (error, docs) {
            if(error == null) {
                console.log('Sucessfully saved new link');
                console.log(docs[0]);
                io.sockets.emit('newLinkCallback', { _id: docs[0]._id, source: docs[0].source, target: docs[0].target });
            } 
        });
    });
});

It can be seen that the code above pretty much achieves the following functions:

  • Creates a couple of MongoDBProviders (which will see more on later)
  • Configures Express
  • Creates the Http server
  • Creates a Socket.IO WebSocket server, and sets up message handlers and callbacks to the clients

I think most of this is self explanatory, but I we will be covering some of it, such as the routes and how the Socket.IO WebSockets stuff works in more detail

Demo App Screens And How They Work

Assuming you have managed to get the demo app up and running you should now be able to browse using the following URL:

Home Route

http://localhost:2000/home: Which should show you something like this:

From here you can fill in a email address (which will be validated as being correctly formatted) and then click enter to proceed to the next screen. I use jQueryUI to show validation errors.

Recall from earlier that this is what this route does

  • It has a GET, which allows the home page to be viewed. The 1st time this route is called some initial seed data will be placed into the application MongoDB database (Providing there is not already some data present)
  • It also has a POST which will accept the entered Email address for a new user, and will save that in the application MongoDB database, and the redirect the user to the /d3Demo route

The Home route is handled by the following code:

/*
 * GET home page : Seed the MongoDB data on 1st GET request
 */

exports.homeGet = function(req, res, commonHelper, personProvider, linkProvider){
        commonHelper.seedData(personProvider, linkProvider, function() {
        res.render('home');
    });
};


/*
 * POSTED home page : Saved its the state in MongodDB
 * and redirect to D3Demo route
 */

exports.homePost = function (req, res, personProvider) {
    var newUserEmail = req.body.email;
    console.log("/Home posted Email :" + newUserEmail);
    personProvider.save({
        email: newUserEmail,
    }, function (error, docs) {
        if(error == null) {
            res.redirect('/d3demo');
        } else {
            res.render('home');
        }
    });
};

I think most of this explained in the comments above, apart from how the data is seeded which is done as follows:

CommonHelper = function () {
};


CommonHelper.prototype.getPersistedDataCount = function (provider, callback) {
    provider.findAll(function (error, docs) {
        var data = 0;
        if (error == null) {
            callback(docs.length);
        }
        else {
            callback(data);
        }
    });
}


CommonHelper.prototype.seedData = function (personProvider, linkProvider, callback) {
    personProvider.findAll(function (error, docs) {
        var data = 0;
        if (error == null) {
            if (docs.length == 0) {
                seedPeople(personProvider, function () {
                    seedLinks(linkProvider, callback);
                });
            } else {
                callback();
            }
        }
        else {
            callback();
        }
    });
}

function seedPeople(personProvider, callback) {
    console.log('No existing users were found so seeding people data');
    personProvider.save(
        [
            { "email": "andy.monks@gmail.com" },
            { "email": "ryanW@gmail.com" },
            { "email": "sarah2008@gmail.com" },
            { "email": "sarahb@hotmail.com" },
            { "email": "sachabarber@gmail.co.uk" }
        ]
        , function (error, docs) {
            if (error != null) {
                console.log("There was an error seeding the data, bugger all can be done....ooops");
            }
            else {
                console.log('There are now ' + docs.length + ' users');
            }
            callback();
        });
}

function seedLinks(linkProvider, callback) {
    console.log('No existing links were found so seeding link data');
    linkProvider.save(
        [
            { "source": "andy.monks@gmail.com", "target": "ryanW@gmail.com" },
            { "source": "andy.monks@gmail.com", "target": "sarah2008@gmail.com" },
            { "source": "andy.monks@gmail.com", "target": "sachabarber@gmail.co.uk" }
        ]
        , function (error, docs) {
            if (error != null) {
                console.log("There was an error seeding the data, bugger all can be done....ooops");
            }
            else {
                console.log('There are now ' + docs.length + ' links');
            }
            callback();
        });
}

exports.CommonHelper = CommonHelper;

The only other thing that really needs further explanation is how the MongoDB database providers work.

This is shown below for the PersonProvider where this code is based on the code found in this article : http://howtonode.org/express-mongodb

var Db = require('mongodb').Db;
var Connection = require('mongodb').Connection;
var Server = require('mongodb').Server;
var BSON = require('mongodb').BSON;
var ObjectID = require('mongodb').ObjectID;

PersonProvider = function (host, port) {
    this.db = new Db('node-mongo-socketDemo', new Server(host, port, { safe:true, auto_reconnect: true }, {}));
    this.db.open(function () { });
};

PersonProvider.prototype.getCollectionSafe = function (callback) {
    this.db.collection('people', { safe: true }, function (error, person_collection) {
        if (error) callback(error);
        else callback(null, person_collection);
    });
};

PersonProvider.prototype.getCollection = function (callback) {
    this.db.collection('people', function (error, person_collection) {
        if (error) callback(error);
        else callback(null, person_collection);
    });
};

PersonProvider.prototype.findAll = function (callback) {
    this.getCollection(function (error, person_collection) {
        if (error) callback(error)
        else {
            person_collection.find().toArray(function (error, results) {
                if (error) callback(error)
                else callback(null, results)
            });
        }
    });
};

PersonProvider.prototype.findById = function (id, callback) {
    this.getCollection(function (error, person_collection) {
        if (error) callback(error)
        else {
            person_collection.findOne({ _id: person_collection.db.bson_serializer.ObjectID.createFromHexString(id) }, 
                function (error, result) {
                    if (error) callback(error)
                    else callback(null, result)
            });
        }
    });
};

PersonProvider.prototype.save = function (people, callback) {
    this.getCollection(function (error, person_collection) {
        if (error) callback(error)
        else {
            if (typeof (people.length) == "undefined")
                people = [people];
            person_collection.insert(people, function () {
                callback(null, people);
            });
        }
    });
};
exports.PersonProvider = PersonProvider;

The LinkProvider is exactly the same code but it deals with a different collection.

D3Demo Route

This is the route that will you will be directed to when you come from the home route. It should be noted that the 1st time you go through the home route some seed data is placed in the Mongo DB database, such that the demo has some nice data to see the 1st time you run it.

You are however completely fine to navigate to this url at any time you like, and it should show you the uber cut down social network graph something like that shown below, which you can add to using the buttons provided

http://localhost:2000/d3demo: Which should show you something like this (click the image for larger version):

Recall from earlier that this is what this route does

  • Has a GET which simply grabs all the previously stored Person and Link objects from the application MongoDB database and shows this in a D3.js Force type (think spring connected) graph layout
  • Has a button to add a new Person, which will launch a jQuery UI dialog to allow the new details to be entered. If entered correctly shall call the Socket.IO "addNewPersonMessage" message to ask the node application server to add a new Person to the application MongoDB database
  • Has a button to add a new Link, which will launch a jQuery UI dialog to allow the new details to be entered. If entered correctly shall call the Socket.IO "addNewLinkMessage" message to ask the node application server to add a new Person to the application MongoDB database
  • Initialises a Socket.IO WebSocket client connection back to the applications Socket.IO WebSocket server, which allows the following messages and callbacks to be sent and received
    • addNewPersonMessage: Message from client to server to add new Person object to application MongoDB database
    • addNewLinkMessage: Message from client to server to add new Link object to application MongoDB database
    • newPersonCallback: Message from server to ALL clients, which allows client(s) to respond to the new Person that has been added. In this demo this simply means adding the new Person to the D3.js Force type (think spring connected) graph layout
    • newLinkCallback: Message from server to ALL clients, which allows client(s) to respond to the new Link that has been added. In this demo this simply means adding the new Link to the D3.js Force type (think spring connected) graph layout

The basic D3Demo route is a simple GET which returns a model which is made up of an array of Person and their count, and Links and their count

/*
 * GET D3 gets a model which has an array of Person and their count, and Links and their count
 */

exports.d3demoGet = function (req, res, commonHelper, personProvider, linkProvider) {
    personProvider.findAll(function (errorPeople, people) {
        if (errorPeople == null) {
            var peopleCount=people.length;
            var peopleJson = JSON.stringify(people);
            linkProvider.findAll(function (errorLinks, links) {
                if (errorLinks == null) {
                    var linksCount=links.length;
                    var linksJson = JSON.stringify(links);
                    res.render('d3demo', 
                    { 
                        peopleCount: peopleCount, 
                        people: peopleJson,  
                        linksCount: linksCount, 
                        links: linksJson  
                    });
                }
                else {
                    res.render('d3demo', 
                        { 
                            peopleCount: peopleCount, 
                            people: peopleJson,  
                            linksCount: 0, 
                            links: JSON.stringify([]),  
                        });
                }
            });
        }
        else {
            res.render('d3demo', 
                { 
                    peopleCount: 0, 
                    people: JSON.stringify([]),  
                    linksCount: 0, 
                    links: JSON.stringify([]),  
                });
        }
    });
};

When the page is rendered it will create a D3.JS force directed graph. Here is the code that creates the graph of the model data.

function createGraph() {

    graph = new myGraph("#graph");

    var people = $.parseJSON($('#people').val());
    peopleCount = people.length;
    for (var i = 0; i < people.length; i++) {
        graph.addNode(people[i]._id, people[i].email);
    }

    var links = $.parseJSON($('#links').val());
    linksCount = links.length;
    for (var i = 0; i < links.length; i++) {
        graph.addLink(links[i].source, links[i].target);
    }
}

function myGraph(el) {

    // Add and remove elements on the graph object
    this.addNode = function (id, email) {
        nodes.push({ "id": id, "email": email });
        update();
    }

    this.removeNode = function (email) {
        var i = 0;
        var n = findNode(email);
        while (i < links.length) {
            if ((links[i]['source'] == n) || (links[i]['target'] == n)) links.splice(i, 1);
            else i++;
        }
        nodes.splice(findNodeIndex(email), 1);
        update();
    }

    this.addLink = function (source, target) {
        links.push({ "source": findNode(source), "target": findNode(target) });
        update();
    }

    var findNode = function (email) {
        for (var i in nodes) { if (nodes[i]["email"] === email) return nodes[i] };
    }

    var findNodeIndex = function (email) {
        for (var i in nodes) { if (nodes[i]["email"] === email) return i };
    }

    // set up the D3 visualisation in the specified element
    var w = $(el).innerWidth(),
            h = $(el).innerHeight();

    var vis = this.vis = d3.select(el).append("svg:svg")
                .attr("width", w)
                .attr("height", h);

    var force = d3.layout.force()
                .gravity(.05)
                .distance(100)
                .charge(-100)
                .size([w, h]);

    var nodes = force.nodes(),
                links = force.links();

    var update = function () {
        var link = vis.selectAll("line.link")
                .data(links, function (d) { return d.source.id + "-" + d.target.id; });
        link.enter().insert("line")
                .attr("class", "link");
        link.exit().remove();
        var node = vis.selectAll("g.node")
                .data(nodes, function (d) { return d.email; });

        var nodeEnter = node.enter().append("g")
                .attr("class", "node")
                .call(force.drag);

        nodeEnter.append("image")
                .attr("class", "circle")
                .attr("xlink:href", "../Images/Friend.png")
                .attr("x", "-8px")
                .attr("y", "-8px")
                .attr("width", "40px")
                .attr("height", "40px");

        nodeEnter.append("text")
                .attr("class", "nodetext")
                .attr("dx", 12)
                .attr("dy", ".35em")
                .text(function (d) { return d.email });
        node.exit().remove();

        force.on("tick", function () {
            link.attr("x1", function (d) { return d.source.x; })
                    .attr("y1", function (d) { return d.source.y; })
                    .attr("x2", function (d) { return d.target.x; })
                    .attr("y2", function (d) { return d.target.y; });

            node.attr("transform", function (d) { return "translate(" + d.x + "," + d.y + ")"; });
        });

        // Restart the force layout.
        force.start();
    }

    // Make it all go
    update();
}

I can't really go into a blow by blow account of how to use D3.Js but it has very good demos and good documentation at http://d3js.org/. If you have not come across D3.Js you should really check it out, as it really it quite a mad library, which has the following features to name a few:

  • jQuery like selectors
  • SVG Based graphics
  • Dynamic layouts

There is also a very good D3.Js tutorial here by Espen Harlinn: http://www.codeproject.com/Articles/523444/D3-crash-course which I recommend you read 

From the d3Demo page you are also able to manipulate your own session, and submit edits back to the server via a WebSocket which will be broadcast to other clients connected to the application's server socket.

Let's continue to have a look at how you can use the WebSockets.

Client Side WebSocket Comms

We have already seen the server side portion of the Socket.IO WebSocket setup which was done as follows:

io.sockets.on('connection', function (socket) {

    socket.on('addNewPersonMessage', function (data) {
        console.log("addNewPersonMessage recieved person (email:'" + data.email + "')");

        personProvider.save({
            email: data.email
        }, function (error, docs) {
            if(error == null) {
                console.log('Sucessfully saved new person');
                console.log(docs[0]);
                io.sockets.emit('newPersonCallback', { _id: docs[0]._id, email: docs[0].email });
            } 
        });
    });
    socket.on('addNewLinkMessage', function (data) {
        console.log("addNewLinkMessage recieved link (source:'" + data.source + "', target:'" + data.target + "')");
        linkProvider.save({
            source: data.source,
            target: data.target
         }, function (error, docs) {
            if(error == null) {
                console.log('Sucessfully saved new link');
                console.log(docs[0]);
                io.sockets.emit('newLinkCallback', { _id: docs[0]._id, source: docs[0].source, target: docs[0].target });
            } 
        });
    });
});

So how does this server side socket code interact with the client side JavaScript code. Well that is all handled within the "socketComms.js" client side code file, where the socket is setup as follows:

$(document).ready(function () {

    var socket = io.connect('http://localhost');
    socket.on('newPersonCallback', onNewPersonCallback);
    socket.on('newLinkCallback', onNewLinkCallback);
}

Add Person

Which will show a jQuery UI dialog which allows a new Person to be added, which if successfully added to the Mongo DB database will be broadcast to everyone using the Node.js Socket.IO package, which provides WebSocket communications between server and client.

It can be seen from this dialog that a new person (for the uber simple social network) requires a email. Essentially assuming that the the user picks the correct data the following code is called which calls the server side socket code, which results in a new Person being persisted to the database. This new Person is then broadcast to ALL connected server socket clients, where they all handle the callback, and add the new Person to their own graphs.

The important line below is the "Socket.Emit(..)" which is the client writing data to the server socket

$("#new-people-dialog").dialog({
    autoOpen: false,
    height: 250,
    width: 500,
    title: 'Enter a email for the new person',
    modal: true,
    buttons: {
        "Add email": function () {
            var isValid = true;
            var email = $('#NewPersonEmail');
            allFields = $([]).add(email);
            allFields.removeClass("ui-state-error");
            isValid = isValid && checkLength(email, 6, 80);
            // From jquery.validate.js (by joern), contributed by Scott Gonzalez:
            //          http://projects.scottsplayground.com/email_address_validation/
            isValid = isValid && checkRegexp(email, ......);
            if (isValid) {
                var emailValue = email.val();
                socket.emit('addNewPersonMessage', { email: emailValue });
                $(this).dialog("close");
            }
        },
        Cancel: function () {
            $(this).dialog("close");
        }
    },
    close: function () {
        allFields.val("").removeClass("ui-state-error");
    }
});

function onNewPersonCallback(data) {
    peopleCount = peopleCount + 1;
    createSummary();
    graph.addNode(data._id, data.email);
}

And here is the relevant server side socket code

socket.on('addNewPersonMessage', function (data) {
    console.log("addNewPersonMessage recieved person (email:'" + data.email + "')");

    personProvider.save({
        email: data.email
    }, function (error, docs) {
        if(error == null) {
            console.log('Sucessfully saved new person');
            console.log(docs[0]);
            io.sockets.emit('newPersonCallback', { _id: docs[0]._id, email: docs[0].email });
        } 
    });
});

Add Link

Which will show a jQuery UI dialog which allows a new Link between two existing people to be added, which is broadcast to everyone using the Node.js Socket.IO package:

It can be seen from this dialog that a new link (for the uber simple social network) requires two existing people (from existing people). Essentially assuming that the the user picks the correct data the following code is called which calls the server side socket code, which results in a new Link being persisted to the database. This new Link is then broadcast to all connected server socket clients, where they all handle the callback, and add the new Link to their own graphs.

The important line below is the "Socket.Emit(..)" which is the client writing data to the server socket

$("#new-links-dialog").dialog({
    autoOpen: false,
    height: 300,
    width: 500,
    title: 'Enter a link between existing people',
    modal: true,
    buttons: {
        "Add link": function () {
            var isValid = true;
            var sourcePeople = $('#sourcePeople');
            var targetPeople = $('#targetPeople');

            allFields = $([]).add(sourcePeople, targetPeople);
            allFields.removeClass("ui-state-error");
            isValid = isValid && checkHasSelection('#sourcePeople option:selected');
            isValid = isValid && checkHasSelection('#targetPeople option:selected');
            if (isValid) {
                var sourcePerson = $('#sourcePeople option:selected').text();
                var targetPerson = $('#targetPeople option:selected').text();
                socket.emit('addNewLinkMessage', { source: sourcePerson, target: targetPerson });
                $(this).dialog("close");
            }
        },
        Cancel: function () {
            $(this).dialog("close");
        }
    },
    close: function () {
        allFields.val("").removeClass("ui-state-error");
    }
});


function onNewLinkCallback(data) {
    console.log('onNewLinkCallback');
    console.log(data);
    linksCount = linksCount + 1;
    createSummary();
    graph.addLink(data.source, data.target);
}

And here is the relevant server side socket code

socket.on('addNewLinkMessage', function (data) {
    console.log("addNewLinkMessage recieved link (source:'" + data.source + "', target:'" + data.target + "')");
    linkProvider.save({
        source: data.source,
        target: data.target
        }, function (error, docs) {
        if(error == null) {
            console.log('Sucessfully saved new link');
            console.log(docs[0]);
            io.sockets.emit('newLinkCallback', { _id: docs[0]._id, source: docs[0].source, target: docs[0].target });
        } 
    });
});

My Final Thoughts

The Good

I have to say I really enjoyed creating this article/small demo application, and I found Node.js to very well thought out, and it does have a great community behind it, and there are many many great packages out there. Generally I would get stuck (yes that happened a few times), but the answer would generally be pretty obvious, and if I really got stuck which I did once, I would find the answer on StackOverflow.com pretty easily. One of the things I really liked about Node.js is just how light weight it is.

So that's the good what about the bad. Sadly there was one bad apple, at least for me there was.

The Bad

One area where I really struggled was using the Jade view engine, which as I have stated uses semantic white space. I would find that I would be using a text editor and putting in what I thought to be quite reasonable spaces, and I would fire the app up, and I would just end up with an Exception being shown in place of my desired HTML contents. When the Exception would state "You can not mix spaces and tabs", to my mind I had not used any tabs. This was really really tricky to work with, which is a shame as Jade itself its mad and very feature rich, and in many ways is light years ahead of other view engines such as Microsoft Razor which is used in ASP MVC. I just have to question the sanity of anything that uses semantic white space.

I guess this could be fixed by using the correct editor, and being more familiar with Jade I guess. Problem was I really could not see what the Exception was talking about.

The Outstanding Questions I Still Have

I still have a few unanswered questions about Node.js, which are out lined below. Feel free to answer these in this articles forum if you feel you have some opinions on them

  1. How do you manage all the related files for a fully Node.Js application, as there is not really any idea that brings it all together in a single project. Sure I could use a nice text editor such as Sublime text editor, but that is still opening 1 file at a time. I am looking for an integrated   IDE that manages a Node.js project. I resorted to creating a dummy VS2010 solution and dragging all my Node.js apps files into it, such that I could manage it all by opening one single file (the VS2010 solution file).
  2. What text editor do people use wih Jade files. As stated above this was a definite sticking point for me.
  3. I am still not clear what constitutes a good/useful/appropriate Node.Js application. Sure Node,js talks about fast, scalable network applications, but when you out HTML views and the like on top of that, it seems to be just like any other HTML solution that supports some sort of web sockets / SignalR (where SignalR is actually using Socket.IO anyway) comms. I guess what I am asking is what is a real god example of an application that would be a good fit for Node.js.

As I say if some of you reading this feel you have answers to these questions, please make them known in this articles forum

That's It

I hope you have enjoyed this article and have got something out of it. I have quite enjoyed writing this one, and think it turned out OK. If you did enjoy it, a vote or a comment are most welcome. 

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Sacha Barber
Software Developer (Senior)
United Kingdom United Kingdom
I currently hold the following qualifications (amongst others, I also studied Music Technology and Electronics, for my sins)
 
- MSc (Passed with distinctions), in Information Technology for E-Commerce
- BSc Hons (1st class) in Computer Science & Artificial Intelligence
 
Both of these at Sussex University UK.
 
Award(s)

I am lucky enough to have won a few awards for Zany Crazy code articles over the years

  • Microsoft C# MVP 2014
  • Codeproject MVP 2014
  • Microsoft C# MVP 2013
  • Codeproject MVP 2013
  • Microsoft C# MVP 2012
  • Codeproject MVP 2012
  • Microsoft C# MVP 2011
  • Codeproject MVP 2011
  • Microsoft C# MVP 2010
  • Codeproject MVP 2010
  • Microsoft C# MVP 2009
  • Codeproject MVP 2009
  • Microsoft C# MVP 2008
  • Codeproject MVP 2008
  • And numerous codeproject awards which you can see over at my blog

Comments and Discussions

 
GeneralMy vote of 5 PinmemberPaul Tait7-Jan-13 15:44 
GeneralRe: My vote of 5 PinmvpSacha Barber7-Jan-13 21:44 
QuestionSome questions PinmemberSuper Lloyd7-Jan-13 13:26 
AnswerRe: Some questions PinmvpSacha Barber7-Jan-13 21:47 
QuestionTo answer some of your questions. PinprotectorPete O'Hanlon7-Jan-13 11:41 
AnswerRe: To answer some of your questions. PinmvpSacha Barber7-Jan-13 12:03 
AnswerRe: To answer some of your questions. PinmvpSacha Barber8-Jan-13 3:52 
GeneralRe: To answer some of your questions. PinprotectorPete O'Hanlon8-Jan-13 4:15 
No probs mate. I do like it.

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

CodeStash - Online Snippet Management | My blog | MoXAML PowerToys | Mole 2010 - debugging made easier

GeneralMy vote of 5 PinmvpSacha Barber7-Jan-13 10:56 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.141015.1 | Last Updated 11 Feb 2013
Article Copyright 2013 by Sacha Barber
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid