Making Dashboards with Dc.js - Part 1: Using Crossfilter.js

Introduction

Dc.js is a JavaScript library used to make interactive dashboards in JavaScript. By clicking and selecting different events in graphs, you can filter the entire dashboard to drill into a particular event.

This is the first of a 4 part series. You can find my other articles here:

• Part 1: Using Crossfilter.js 
• Part 2: Graphing
• Part 3: Tips and Tricks  
• Part 4: Style

Background

In order to do this, DC.js relies on two other JavaScript plugins/libraries: D3.js and Crossfilter js.

D3.js is the evolution of Protovis. Up until a few years ago, most browser-based graphs were static (Protovis) or based on a non-JavaScript plugin like Flash or Java, which hides the underlying code making it impossible to alter with new functionality or have it work without the plug-in. D3.js, on the other hand, is done in the browser's native JavaScript and SVG, so you can see how it is manipulating objects inside the DOM. With D3.js, it is now much easier to make graphs interactive.

While D3.js allows you to make really cool graphs, it isn't a graphing library. D3.js will build and manipulate co-ordinate systems, axises and shapes; but it doesn't know what a bar chart or a pie chart is. This is where Dc.js comes in. Dc.js defines line graphs, bar and pie charts and uses D3.js's objects to build them. This makes it much easier to focus on what you want to display instead of generating the display itself.

Crossfilter.js is a JavaScript plug-in used to slice and dice JavaScript arrays. This allows Dc.js to easily manipulate the datatable that the graphs use, so they can refresh with the filtered data. The example on the Crossfilter website was the actual inspiration for the dc.js library itself.

Using the Code

Now that we have the background, we can start to write some code. With this first art, we're going to start off with how Crossfilter works because most of the actual dashboard code is manipulating the data for the charts. Once you have a good understanding of how that works, the actual graphing is pretty simple. I'm going to try to cover several different scenarios, so you can hopefully avoid some of the pitfalls as you start to use it.

Download the Crossfilter.js file from GitHub and include it in your HTML page. For these examples, I'm going to use the raw GitHub source for a reference.

<script type="text/javascript" src="https://rawgithub.com/NickQiZhu/dc.js/master/web/js/crossfilter.js"></script>

We'll first need some data. This data was pulled from the Crossfilter API documentation:

var data = [
  {date: "2011-11-14T16:17:54Z", quantity: 2, total: 190, tip: 100, type: "tab"},
  {date: "2011-11-14T16:20:19Z", quantity: 2, total: 190, tip: 100, type: "tab"},
  {date: "2011-11-14T16:28:54Z", quantity: 1, total: 300, tip: 200, type: "visa"},
  {date: "2011-11-14T16:30:43Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T16:48:46Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T16:53:41Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T16:54:06Z", quantity: 1, total: 100, tip: 0, type: "cash"},
  {date: "2011-11-14T16:58:03Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T17:07:21Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T17:22:59Z", quantity: 2, total: 90, tip: 0, type: "tab"},
  {date: "2011-11-14T17:25:45Z", quantity: 2, total: 200, tip: 0, type: "cash"},
  {date: "2011-11-14T17:29:52Z", quantity: 1, total: 200, tip: 100, type: "visa"}
]; 

We'll make our Crossfilter instance.

var ndx = crossfilter(data);

For our first example, we'll setup a filter using one of the integer columns. Say we want to get all the transactions with a total equal to 90. To do this, we need to setup a dimension.

var totalDim = ndx.dimension(function(d) { return d.total; });   

Now we can start to filter it. If we wanted to find all the totals equal to 90, we can do the following:

var total_90 = totalDim.filter(90); 

To see the result, we can print out the total_90 variable to the console.

print_filter("total_90"); 

This prints out the following. I'll mark the webconsole data in black with white writing:

"total_90(6) = [
	{"date":"2011-11-14T17:22:59Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T17:07:21Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:58:03Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:53:41Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:48:46Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:30:43Z","quantity":2,"total":90,"tip":0,"type":"tab"}
]"  

### Tip ###

Since we want to be able to see if our filters are working correctly, I've created a small function to printout the data to the webconsole ("Tools" > "Web Developer" > "Web Console" in Firefox).

function print_filter(filter){
	var f=eval(filter);
	if (typeof(f.length) != "undefined") {}else{}
	if (typeof(f.top) != "undefined") {f=f.top(Infinity);}else{}
	if (typeof(f.dimension) != "undefined") {f=f.dimension(function(d) { return "";}).top(Infinity);}else{}
	console.log(filter+"("+f.length+") = "+JSON.stringify(f).replace("[","[\n\t").replace(/}\,/g,"},\n\t").replace("]","\n]"));
} 

### End Tip ###

Each of these short hand filters has a long hand equivalent. For filter(90), it is the same as using filterExact(90):

var total_90 = totalDim.filterExact(90); 

print_filter("total_90"); 

"total_90(6) = [
	{"date":"2011-11-14T17:22:59Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T17:07:21Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:58:03Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:53:41Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:48:46Z","quantity":2,"total":90,"tip":0,"type":"tab"},
	{"date":"2011-11-14T16:30:43Z","quantity":2,"total":90,"tip":0,"type":"tab"}
]"   

If we want to filter a range from 90 to 100 inclusive, we'd put the parameter in brackets. Since we want to include 100 in our filter, we'll need to have it go to 101. This is the same as total Dim.filterRange([90,101]);

var total_90_101= totalDim.filter([90,101]); 

print_filter("total_90_101");  

"total_90_101(7) = [
	{"date":"2011-11-14T16:54:06Z","quantity":1,"total":100,"tip":0,"type":"cash"},	
	{"date":"2011-11-14T17:22:59Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T17:07:21Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:58:03Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:53:41Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:48:46Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:30:43Z","quantity":2,"total":90,"tip":0,"type":"tab"}
]" 

You can even get a little more fancy like only grabbing items divisible by 3. This is the same as totalDim.filterFunction(function(d) { if (d%3===0)return d; } );

var total_3= totalDim.filter(function(d) { if (d%3===0) {return d;} } );  

print_filter("total_3"); 

"total_3(7) = [	
	{"date":"2011-11-14T16:28:54Z","quantity":1,"total":300,"tip":200,"type":"visa"},
	{"date":"2011-11-14T17:22:59Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T17:07:21Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:58:03Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:53:41Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:48:46Z","quantity":2,"total":90,"tip":0,"type":"tab"},	
	{"date":"2011-11-14T16:30:43Z","quantity":2,"total":90,"tip":0,"type":"tab"}
]" 

That was pretty easy. Filtering on numbers is pretty straightforward unless you have an entry which is NaN. This will completely confuse crossfilter, so make sure there are no NaNs in your dataset.

Moving onto strings, if we wanted to find all the entries where people used visa, again, we need to first create a dimension, this time on the type, then we can do our filter.

var typeDim  = ndx.dimension(function(d) {return d.type;});

var visa_filter = typeDim.filter("visa"); 

print_filter("visa_filter"); 

"visa_filter(2) = [	
	{"date":"2011-11-14T17:29:52Z","quantity":1,"total":200,"tip":100,"type":"visa"},
	{"date":"2011-11-14T16:28:54Z","quantity":1,"total":300,"tip":200,"type":"visa"}
]"

If we wanted to do the same thing with cash, again, pretty easy.

var cash_filter = typeDim.filter("cash");  

print_filter("cash_filter");

"cash_filter(2) = [	
	{"date":"2011-11-14T17:25:45Z","quantity":2,"total":200,"tip":0,"type":"cash"},	
	{"date":"2011-11-14T16:54:06Z","quantity":1,"total":100,"tip":0,"type":"cash"}
]"  

We can even sum up our total column for just the cash entries using the ReduceSum function. Ok, this is where it gets a little tricky. Previously, we were filtering on the dimension. You would think that you'd use the reduceSum function on the filtered data. That is not the case. If we do a ReduceSum on the filtered data through the group function, it won't observe the current filter and will give you back the totals per type in key value format. This kindof makes sense and is pretty handy for dc.js, but not when you want to try to access the data for cash.

var total = typeDim.group().reduceSum(function(d) {return d.total;});

print_filter("total");

"total(3) = [
	{"key":"tab","value":920},
	{"key":"visa","value":500},
	{"key":"cash","value":300}
]"

Instead to get the total for cash, you have to do a Groupall on the crossfilter object itself, which observes all filters; so that when we do a ReduceSum, we get the sum of the total column for the current filter.

var cash_total = ndx.groupAll().reduceSum(function(d) {return d.total;}).value() 

console.log("cash_total="+cash_total);  

"cash_total=300" 

So if the crossfilter object observes all filters, how come it didn't observer the visa filter when we decided to filter on cash? Well ... it's just kind of quirky. If you try to do a filter for cash and visa, it still has the cash filter applied.:

var cash_and_visa_filter = typeDim.filter(function(d) { if (d ==="visa" || d==="cash") {return d;} });  

print_filter("cash_and_visa_filter");

"cash_and_visa_filter(2) = [
	{"date":"2011-11-14T17:25:45Z","quantity":2,"total":200,"tip":0,"type":"cash"},
	{"date":"2011-11-14T16:54:06Z","quantity":1,"total":100,"tip":0,"type":"cash"}
]" 

We need to first clear all filters and then it will work. It's good practise to always clear filters before starting another one.

typeDim.filterAll()

var cash_and_visa_filter = typeDim.filter(function(d) { if (d ==="visa" || d==="cash") {return d;} });

print_filter("cash_and_visa_filter");

"cash_and_visa_filter(4) = [
	{"date":"2011-11-14T17:29:52Z","quantity":1,"total":200,"tip":100,"type":"visa"},
	{"date":"2011-11-14T16:28:54Z","quantity":1,"total":300,"tip":200,"type":"visa"},
	{"date":"2011-11-14T17:25:45Z","quantity":2,"total":200,"tip":0,"type":"cash"},
	{"date":"2011-11-14T16:54:06Z","quantity":1,"total":100,"tip":0,"type":"cash"}
]" 

You've now seen the basics of what crossfilter can do. The next article will showcase on using it with dc.js.