Click here to Skip to main content
15,439,290 members
Articles / Web Development / Node.js
Posted 25 Dec 2013

Tagged as


18 bookmarked

How to scrape data from the Web using Node.js

Rate me:
Please Sign up or sign in to vote.
4.55/5 (11 votes)
6 Jan 2014CPOL2 min read
This article shows you how to fetch data from the Web using the powerful Node.js


Using Node.js , you can do what you want like a website for chat , a Social Network Like LinkedIn and Facebook and also you can fetch data from The Web.


In the past, I have wrote this post on the different options you can use to scrape data from the Web using for the HtmlAgilityPack in .Net Development Environment. So you can do the same functionality using the powerful Node.Js .

So Node.js is a platform built on Chrome's JavaScript Runtime for easily building fast, scalable network application. Node.js uses an event-driven, non-blocking Input Output model that make it lightweight and  efficient, perfect for data-intensive real-time applications that run across distributed devices. you can download it here : . 

Using the code

You Will need to install Node.js of course and on top it 3 packages:

  1. npm install request: we could work with the URLs in an easy way.
  2. npm install cheerio: Cheerio is a jQuery for the server side.
  3. npm install fs: We use this Package to make files.   

After you finish the installation process, you should start a JavaScript File that contains our Code. So the first thing you must do is call the modules we need in our application: 

var request= require('request'),
	cheerio = require('cheerio'),
	fs = require('fs'), 
	urls= []; // urls is a variable to store the urls of the pictures. 

After that we must add the main call to fetch data from the website, parse it and work on it . In this case, we will fetch the location of the picture founded on as shown below:

request('', function(err,resp,body)
	if(!err && resp.statusCode == 200)
		var $ = cheerio.load(body);
		$('a.title', '#siteTable').each(function(){
			var url = this.attr('href');

Now let's do something with the data we have : we will store the picture founded on the Directory called"img" that you must create it inside the directory of your work. 

                for (var i=0;i<urls.length;i++)
			request(urls[i]).pipe(fs.createWriteStream('img/' + i +'.jpg'));
	// now we will close the curly braces and end our program	

Well done. Now, we have to type on The Node.js Command Prompt : Node Scrapping.js.

Now, you have all the pictures of the websites stored on your directory. 

Any comments are welcome! 


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
Tunisia Tunisia
Microsoft Certified Professional, Big Data and Cloud Architect.
Mail :

Comments and Discussions

QuestionThe script is doing nothing Pin
Amir Khan16-Mar-15 22:06
MemberAmir Khan16-Mar-15 22:06 
Questionscrape wole website Pin
Member 1089203825-Aug-14 7:48
MemberMember 1089203825-Aug-14 7:48 
Questionweb scrapping Pin
Member 1089203814-Aug-14 22:01
MemberMember 1089203814-Aug-14 22:01 
QuestionWhat is the actual use of Node.js Pin
Tridip Bhattacharjee6-Jan-14 18:59
professionalTridip Bhattacharjee6-Jan-14 18:59 
AnswerRe: What is the actual use of Node.js Pin
Hadrich Mohamed26-Feb-14 22:55
professionalHadrich Mohamed26-Feb-14 22:55 
GeneralRe: What is the actual use of Node.js Pin
Tridip Bhattacharjee27-Feb-14 20:15
professionalTridip Bhattacharjee27-Feb-14 20:15 
AnswerRe: What is the actual use of Node.js Pin
DaveAuld31-May-14 2:27
professionalDaveAuld31-May-14 2:27 
Question"npm install fs" does not work Pin
mhn21728-Dec-13 0:06
Membermhn21728-Dec-13 0:06 
AnswerRe: "npm install fs" does not work Pin
Hadrich Mohamed28-Dec-13 3:36
professionalHadrich Mohamed28-Dec-13 3:36 
Generalfs works Pin
mhn2172-Jan-14 2:14
Membermhn2172-Jan-14 2:14 
Questionno fs package Pin
ej8ej27-Dec-13 12:11
Memberej8ej27-Dec-13 12:11 
AnswerRe: no fs package Pin
pipiscrew28-Dec-13 0:23
Memberpipiscrew28-Dec-13 0:23 
GeneralRe: no fs package Pin
Hadrich Mohamed28-Dec-13 3:53
professionalHadrich Mohamed28-Dec-13 3:53 
GeneralRe: no fs package Pin
ej8ej29-Dec-13 9:24
Memberej8ej29-Dec-13 9:24 
Questionuse nodeJSgui Pin
pipiscrew27-Dec-13 0:37
Memberpipiscrew27-Dec-13 0:37 
QuestionHow to get only html without images Pin
Sunasara Imdadhusen25-Dec-13 23:33
professionalSunasara Imdadhusen25-Dec-13 23:33 
AnswerRe: How to get only html without images Pin
Hadrich Mohamed26-Dec-13 1:30
professionalHadrich Mohamed26-Dec-13 1:30 
Thanks for your Message.
So to retrieve some data like title , description you should know where the data are stored in ? class or id ?
//to retrive all website
var $ = cheerio.load(body);
//to scrape just the title
$('a.title', '#siteTable').each(function(){
//to store just the content of "href" of a.title
var url = this.attr('href');
you can make your changes to get just the mobile names , prices , and url
QuestionNode.js is still confusing to me Pin
Chicken9925-Dec-13 21:56
MemberChicken9925-Dec-13 21:56 
GeneralRe: Node.js is still confusing to me Pin
Gebbetje4-Jan-14 4:15
MemberGebbetje4-Jan-14 4:15 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.