Build a Demographic Data Visualization Tool Based On d3.js

Introduction

If you're interested in data analysis and data visualization, you may have already stumbled across a tool called Gapminder. Gapminder intends to provide statistical data (mostly from UN organizations) and combines it with nice animation features. The initiator of Gapminder is Swedish professor Hans Rosling, there is an (almost legendary) TED talk of him presenting Gapminder, it is definitely worth a watch if you haven't done it so far.

Gapminder was originally implemented in the year 2005 and it had some nice features that were not so easy to realize back then (at least not with JavaScript). Luckily, nowadays a lot of awesome open JavaScript libraries make the life of web developers a lot easier - and faster. And this is also the purpose of this tutorial: A proof of concept how easy we can implement a data visualization tool when we have the right libraries and APIs for it. Therefore, this tutorial may focus on demographic data visualization, but it can simply be modified for any kind of visualization once the concept is clear. But now, let's start to build an online tool similar to Gapminder!

Before You Start...

Make sure that you have a basic understanding of JavaScript, jQuery, how REST interfaces work and how to read JSON data objects. These are prerequisites for this tutorial. You can find a nice tutorial about jQuery here, some basic information about REST can be found here.

Get the Data

One of the most important questions when it comes to demographic data visualization is the question, where to get the data from. There are literally thousands of APIs from different organizations that offer data, for this tutorial we're going to use a service called INQStats. There are several reasons why I'm using especially this API:

1. It is free to use without any restrictions
2. It gathers demographic data from different trusted sources (e.g. UN Data) and returns it in an unique format
3. It uses a REST API and returns JSON data - this makes it easy to use in our JavaScript/AJAX implementation
4. I'm the author, so I know what is going on behind the scenes and I can adopt the API to my needs if necessary.

If you don't want to use INQStats, you are free to choose an alternative API. Take a look at the API of the Worldbank or the API of the United Nations.

Getting data with INQStats is quite simple: Just register for an API Key here (you'll get it immediately) and start making requests. For example, if you want to receive the population of Australia for the last five years, enter the following request in a browser address bar and press return (don't forget to replace the api_key parameter with your own key):

http://inqstatsapi.inqubu.com/?api_key=ADDKEYHERE&data=population&countries=au

The following code will be returned:

[
  {
    "countryCode":"au",
    "countryName":"Australia",
    "population":[
	  {
	    "year":"2014",
	    "data":"23490736"
	  },{
	    "year":"2013",
	    "data":"23130900"
	  },{
	    "year":"2012",
	    "data":"22683600"
	  },{
	    "year":"2011",
	    "data":"22323900"
	  },{
	    "year":"2010",
	    "data":"22065300"
	  }
	]
  }
]

This is nice when we only need data for a specific country, but in fact, we want to compare all countries of the world. Luckily, this is also possible. If you submit the following HTTP request, an array of JSON objects is returned, containing all countries of the world and the gross domestic product per capita for the year 2014:

http://inqstatsapi.inqubu.com/?api_key=ADDKEYHERE&cmd=getWorldData&data=gdp_capita&year=2014&addRegion=true

Note: If you don't add the year parameter, the newest data will be returned.

We can now write a simple function that takes two data keys (x- and y-axis) and a year as input parameters, returns a list of all countries including said two data keys and the population of the country and saves it in a variable called dataset. With jQuery, this is pretty straight forward:

var dataset = [];
function getData(key1, key2, year) {
	$.ajax({
		url: 'http://inqstatsapi.inqubu.com?api_key=ADDKEYHERE&cmd=getWorldData&data=' + 
		key1 + ',' + key2 + ',population&year=' + year + '&addRegion=true',
		complete: function(response) {
			dataset = JSON.parse(response.responseText);
                },
                error: function(e) {
                    alert("Data could not be loaded from URI\n" + e.statusText);
                },
	});
	return false;
}

Let's Visualize It!

Plain data itself is boring, so let's make some fancy graphics to bring life to the textual stuff! One of the most powerful libraries to do so in JavaScript is d3.js. D3.js supports us when we want to draw and manipulate charts, it is very flexible and can be customized easily - that's why d3.js is currently one of the most popular frameworks for data visualization in the web.

The final result should look like this:

A live demo and the source code can be found here. We'll now go through the most relevant parts of the code to see what is going on! The chart itself will be placed in a container <div id="container"></div>. To add a SVG chart to this container, we use:

var svg = d3.select("#container").append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
    .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

D3 has its own selector function d3.select(). If the passed string starts with #, the element with the specific ID will be selected. Afterwards, a svg tag is added to the container and the function .attr() is called multiple times. .attr() is very powerful and can be used in different ways: The first parameter states to which object or variable a change should apply, the second parameter states how the change looks like. If the second parameter is a function, a change will apply to all elements that were selected. In our case, it's simpler: We set the width, the height and the margin of the chart here. (Note: Of course, width, height and the margins must be declared first). Furthermore, with .append("g"), we add a <g> tag to the SVG - this is used for shape grouping.

As you may have noticed, d3 heavily makes use of method chaining. This means, the result of a previous function is passed to the next function smoothly by concatenating the functions.

Next, we take a look at the getData function:

function getData(key1, key2, year) {
    $.ajax({
        url: 'http://inqstatsapi.inqubu.com?api_key=ADDKEYHERE&cmd=getWorldData&data=' + 
        key1 + ',' + key2 + ',population&year=' + year + '&addRegion=true',
        complete: function(response) {
            var dataObj = JSON.parse(response.responseText);
            if (dataObj.type != undefined && dataObj.type == "error") {
                alert("Error: " + dataObj.msg)
            } else {
                dataset = dataObj;
                update();
            }
        },
        error: function(e) {
            alert("Data could not be loaded from URI\n" + e.statusText);
        },
    });
    return false;
}

We reused the code from above and first added some error handling: When the JSON object that is returned from the INQStats API contains a variable called type and the value is "error", we know something went wrong while trying to get data. Therefore, we'll show the error message in msg by using a simple alert message box. If everything went well, we'll save the received data in a variable called dataset and call the update() function.

In the first part of the update function, we parse all values in our dataset to numbers:

dataset.forEach(function(d) {
    d[xValue] = +d[xValue];
    d[yValue] = +d[yValue];
});

In this case, d represents the current object (the current country object in our dataset array), xValue is the currently selected data key of the x-axis, the same applies for the y-axis. Afterwards, we update our axis:

xScale.domain([d3.min(dataset, function(d) {
    return d[xValue];
}), d3.max(dataset, function(d) {
    return d[xValue];
})]);

yScale.domain([d3.min(dataset, function(d) {
    return d[yValue];
}), d3.max(dataset, function(d) {
    return d[yValue];
})]);

xScale represents the linear scale for the x-axis. The range of the axis can be modified by calling the .domain() function. The first argument is the minimum value, the second argument the maximum value. With d3.min(), we get the smallest value in our dataset, with d3.max() the largest.

Next, we're going to draw our dots. First, we select all circle elements in our SVG graph (they are not added yet but we'll come to this in a second) and bind the dataset to the selection. Then, we use .enter() in combination with .append() on the selection to add those new circle elements, that are not presented in the selection (since none are there, all circles are newly created). Afterwards, we set the coordinates for our points with cx and cy and the radius with r. We make the size dependant of the population of a country, so: the higher the population, the bigger the dot. We also set the ID of each circle element to the name of the country.

var circles = svg.selectAll("circle").data(dataset);

circles.enter().append("circle")
    .attr({
        cx: function(d) {
            return xScale(d[xValue]);
        },
        cy: function(d) {
            return yScale(d[yValue]);
        },
        r: function(d) {
            if (d.population < 1000000) {
                return 2;
            }
            [...]
        },
        id: function(d) {
            return d.countryName;
        }
    })
    .style("stroke", "black")
    .style("fill", function(d) {
        return color(d.region);
    });

Add a Slider and a Play Button to Visualize Yearly Changes

The most spectacular feature in Gapminder is the animation of data changes depended on the year. We're adding a similar functionality, a slider that can be used for year selection and a play button that causes the slider to jump ahead year by year. The function which is triggered when <button name="play" id="play">Play</button> is clicked:

var running = false;
var timer;
$("#slider").value = 2010;
$("#play").on("click", function() {
    var maxstep = 2015,
        minstep = 2009;
    if (running == true) {
        $("#play").html("Play");
        running = false;
        clearInterval(timer);
    } else if (running == false) {
        $("#play").html("Pause");
        sliderValue = $("#slider").val();
        timer = setInterval(function() {
            if (sliderValue >= maxstep) {
                clearInterval(timer);
                return;
            }
            sliderValue++;
            getData(xValue, yValue, sliderValue);
            setTimeout(function() {
                $("#slider").val(sliderValue);
                $('#range').html(sliderValue);
                $("#slider").val(sliderValue);
                year = $("#slider").val();

            }, 500)
        }, 3000);
        running = true;
    }
});

When there is currently no animation running, a new interval is created and getData() is called every 3 seconds. sliderValue is the year and counts up until the maximum year (2015) is reached. There is a delay of 0.5 seconds between calling the data function and updating the values in the DOM. The reason for this is to compensate delays between sending a request and getting data and making it look more "real time" for the users.

Conclusion

The purpose of this tutorial was to show how easy and fast a simple data visualization tool can be built nowadays. Hopefully, the information provided was useful for you, just drop a comment if something is not clear, wrong or if you have additional questions. Thank you and all the best!