Click here to Skip to main content
15,908,175 members
Articles / Web Development / HTML

Screen scraping using YQL and AJAX

Rate me:
Please Sign up or sign in to vote.
5.00/5 (5 votes)
30 Sep 2013CPOL5 min read 53.4K   1K   23   8
A simple application to scrap HTML data in JSON format.


Web scraping always plays a negative role in web-development, but in some cases it is very important. jQuery is greatly helpful for cross-domain scraping and a bunch of examples are available too . Web scraping has been influenced by Yahoo Query Language(YQL). This article is going to provide a basic overview of web-scraping using jQuery and YQL. To represent data I have also used Mustache as HTML template so, I will also provide a short overview of Mustache.js here.


If you are familiar with HTML you will understand this article easily. Simple basic knowledge of  JavaScript and AJAX will do. Don’t worry, I have added some useful links to make you clear.

YQL Overview

YQL is a SQL-like syntax and can be used to work with different APIs. YQL is popular because of its faster response. Details overview can be found here

To get HTML off a page, YQL has different methods called ‘Tables’. Mostly highlighted are given below with example…

In my simple project I’ll be using HTML table.

Querying With YQL Console

Now, let’s become familiar with YQL Console. In the following example we are going to scrap data from gsmarena. We’ll search in gsmarena using phone manufacturer’s name and scrap the result. So,let’s go to gsmarena and see the data we are going to fetch as following

  1. Search using any phone manufacturer’s name, for example, ‘nokia’. 
  2. We have got the search result page. Copy the page’s URL (
  3. Now, let’s go to YQL console and run query to get the result of the mentioned url. (You need to be logged in in Yahoo!)
    1. Go to YQL Console.
    2. In the textbox write the query “select * from html where url=""”.
    3. To get the data in JSON format select JSON and press the ‘TEST’ Button.
  4. That’s all, the result is in the result box. Don’t worry if you do not understand the result, it has scrapped full page ( in JSON format.

  6. But we do not need the full page, we will take only the results generated by the search. To do this, we need the XPATH of the resultant div.
  7. To get the XPATH right click on the page( and copy the XPATH as shown in the picture. 
  8. Now go to YQL Console and run the query with the XPATH again. And now our result is only containing the search result’s div.  
  9. To khow more about XPATH you can follow this w3schools tutorial.

  10. Lastly, to Access YQL API using ajax we need the ‘REST QUERY’. Simply copy the ‘REST QUERY’.
If we see the ‘REST QUERY’ closely we will find three portions*%20from%20html%20where%20url%3D%22http%3A%2F%2Fwww.'%2Fhtml%2Fbody%2F
•   Url to access YQL API (
•   Our full query. Here it is encoded because uri should not contain any whitespace. 
•   Data format and other components (&format=json&diagnostics=true&callback=)


  •  Jquery : In this project JQuery is used for two purposes…

    1. To run ajax request to get data from YQL
    2. To display data using JqueyCycle. To get this feature in just add this script :
      <script src="
        2.9999.8/jquery.cycle.all.min.js" type="text/javascript"></script>

    For both purposes I am using jquery -v1.9.1. You can download any Latest version from jQuery.

  • Json2.js : It is very important to know that From YQL using ajax we get data in JSON format. Json2.js is very helpful to handle this JSON data. Download Json2.js from here and include in your project. To know more about JSON you can go here
  • jsonpath-0.8.0.js : When we want to get the specific result from the received JSON data, It is important to query among the divs, tables etc. jsonpath stands for this purpose. Get any latest version of jsonpath from here and include it in your project.
  • mustache.js : The last requirement for this project to run is mustache.js. Mustache is a “logic-less” template syntax. It is very helpful for decoupling HTML markups from data. Mustache is implemented in different languages: Ruby, JavaScript, Python, PHP, Perl, Objective-C, Java, .NET, Android, C++, Go, Lua, Scala, etc. Mustache.js is the JavaScript implementation. Get mustache.js from here and this one is a helpful tutorial for mustache.

Using the code  

Now our environment is ready to work on. If you face any trouble rearrange it as following

<script src="scripts/jquery-1.9.1.min.js" type="text/javascript"></script>
    <!--jquery cycle library to slide the results-->
    <script src="//" type="text/javascript"></script>
    <!--mustache.js template for javascript -->
    <script src="scripts/mustache.js" type="text/javascript"></script>
    <!--json2.js is to work with json data -->
    <script src="scripts/json2.js" type="text/javascript"></script>
    <!--jsonpath is helpful for querying json data -->
    <script src="scripts/jsonpath-0.8.0.js" type="text/javascript"></script>

So, our task is simple. We’ll take manufacturer’s name as user input, scrap the result from using YQL API and will receive the result performing an AJAX request. So, let’s start…

  • To take the input, we need a text box. A button click event is going to fire a JavaScript function named GetResult(). Also, a div is used to hold the entire result. So, the HTML markup is as following…

        <input id="valueText" type="text" />
        <button type="button" onclick="GetResult()">Get Result</button>
        <div id="speakerbox" style="float:left">
            <a href="#" id="prev_btn">&laquo;</a>
            <a href="#" id="next_btn">&raquo;</a>
            <div id="carousel"></div>
  • JavaScript’s GetResult() function is fetching scrapped data using AJAX. Get user input from textbox.
  • JavaScript
    var item = $('#valueText').val();
  • To build the query first examine the query where the user input is appended as shown in the picture below. 
  • So, we’ll simply append the user input in the query.

    var query = "SELECT * FROM html WHERE url=" + '"' + 
      "" + item + '"' + 
      " and xpath='/html/body/div/div[2]/div[2]/div/div[2]/div[2]'";
  • For now, we will ignore cacheBuster for simplicity. And the rest query for the AJAX URL is:
  • JavaScript
    var url = '' + 
      encodeURIComponent(query) + '&format=json&_nocache=' + cacheBuster;
  • Now, we have everything to access YQL API using AJAX. So, run the AJAX request and get the scrapped data.
    window['wxCallback'] = function (data) {
            ParseData(data); // To show the result
            url: url,
            dataType: 'jsonp',
            cache: true,
            jsonpCallback: 'wxCallback'
  • When the AJAX request performs successfully we need to call ParseData(data) function with the data as its parameter. This function is simply parsing the data and showing it inside the carousel using mustache template.
  • C#
    function ParseData(data) {
        var result = jsonPath(data, "$.query.results[*][*]");
        var html = "";
        for (i = 0; i < result.length; i++) {
            var template = $('#speakerstpl').html();
            html += Mustache.to_html(template, result[i]);
                    fx: 'fade',
                    pause: 1,
                    next: '#next_btn',
                    prev: '#prev_btn',
                    speed: 500,
                    timeout: 10000

    Here, jsonPath is finding the specific ‘li’ which one in containing the data using the provided query.

  • Now, run the app and see the scrapped result in the carousel. 

Points of Interest

No doubt, web-scrapping is an interesting job to do. It is much more interesting with YQL, I think.


  • 28th September, 2013.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
Software Developer Cefalo
Bangladesh Bangladesh
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

QuestionViewing the data in a spreadsheet Pin
JacobLux29-Jun-15 15:22
JacobLux29-Jun-15 15:22 
Generalre:Screen scraping using YQL and AJAX Pin
Member 107920303-May-14 3:10
Member 107920303-May-14 3:10 
BugInvalid jQuery cycle url Pin
Muhammad Umair Irshad10-Oct-13 1:14
Muhammad Umair Irshad10-Oct-13 1:14 
GeneralRe: Invalid jQuery cycle url Pin
Paul Shovan10-Oct-13 8:34
Paul Shovan10-Oct-13 8:34 
Thanks Muhammad Umair Irshad. I'll solve my error
QuestionI don't think that word means what you think it means Pin
B. Clay Shannon30-Sep-13 7:13
professionalB. Clay Shannon30-Sep-13 7:13 
AnswerRe: I don't think that word means what you think it means Pin
Paul Shovan30-Sep-13 8:48
Paul Shovan30-Sep-13 8:48 
GeneralRe: I don't think that word means what you think it means Pin
B. Clay Shannon30-Sep-13 8:51
professionalB. Clay Shannon30-Sep-13 8:51 
GeneralRe: I don't think that word means what you think it means Pin
Duncan Edwards Jones28-Mar-14 5:46
professionalDuncan Edwards Jones28-Mar-14 5:46 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.