.NET Google Search REST API Integration

• Download source code - 38.1 KB

Introduction

This article demonstrates how to integrate the Google Search REST API into a .NET project using managed code (C#). The REST API requires no license key, and queries are unlimited.

The article also covers the deserialization of a JSON response into .NET objects, and workarounds for some of the restrictions / features that come with the use of the API.

Queries to the REST API can be made in any web browser by entering the URL and search terms in the address bar. A call such as the following...

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%Day

returns...

{"responseData": 
  {"results":[{"GsearchResultClass":"GwebSearch", 
   "unescapedUrl":"http://www.earthday.net/", 
   "url":"http://www.earthday.net/", 
   "visibleUrl":"www.earthday.net", 
   "cacheUrl":"http://www.google.com/search?
          q\u003dcache:szJHSzSCm38J:www.earthday.net", 
   "title":"\u003cb\u003eEarth Day\u003c/b\u003e Network", 
   "titleNoFormatting":"Earth Day Network",
   "content":"Get information on \u003cb\u003eEarth 
       Day\u003c/b\u003e events, activities and actions, and learn how you 
       can join the \u003cb\u003eEarth Day\u003c/b\u003e Network 
       and Green Generation to make a difference 
       in \u003cb\u003e...\u003c/b\u003e"},
   {"GsearchResultClass":"GwebSearch",
    "unescapedUrl":"http://en.wikipedia.org/wiki/Earth_Day",
    "url":"http://en.wikipedia.org/wiki/Earth_Day",
    "visibleUrl":"en.wikipedia.org",
    "cacheUrl":"http://www.google.com/search
          ?q\u003dcache:57bkwmGRNFkJ:en.wikipedia.org",
    "title":"\u003cb\u003eEarth Day\u003c/b\u003e - Wikipedia, 
      the free encyclopedia","titleNoFormatting":"Earth 
      Day - Wikipedia, the free encyclopedia",
    "content":"\u003cb\u003eEarth Day\u003c/b\u003e 
       is celebrated in the US on April 22 
       and is a day designed to inspire 
       awareness and appreciation for 
       the Earth\u0026#39;s environment. 
       \u003cb\u003e...\u003c/b\u003e"},
    {"GsearchResultClass":"GwebSearch",
    "unescapedUrl":"http://www.epa.gov/earthday/",
    "url":"http://www.epa.gov/earthday/",
    "visibleUrl":"www.epa.gov",
    "cacheUrl":"http://www.google.com/search?
             q\u003dcache:3IS9GTb-r0IJ:www.epa.gov",
    "title":"\u003cb\u003eEarth Day\u003c/b\u003e | US EPA",
    "titleNoFormatting":"Earth Day | US EPA",
    "content":"Includes history of the celebration, 
           listing of EPA-sponsored events, and other   resources."},
    {"GsearchResultClass":"GwebSearch",
     "unescapedUrl":"http://holidays.kaboose.com/earth-day/",
     "url":"http://holidays.kaboose.com/earth-day/",
     "visibleUrl":"holidays.kaboose.com",
     "cacheUrl":"http://www.google.com/search?
       q\u003dcache:v3MWPNh2yBkJ:holidays.kaboose.com",
     "title":"\u003cb\u003eEarth Day\u003c/b\u003e 2009: 
       Crafts, Environmental Games, and 
       Recycling \u003cb\u003e...\u003c/b\u003e",
     "titleNoFormatting":"Earth Day 2009: 
       Crafts, Environmental Games, and Recycling ...",
     "content":"Jun 1, 2009 \u003cb\u003e...\u003c/b\u003e 
       \u003cb\u003eEarth Day\u003c/b\u003e is a special day to learn 
       about to take care of the planet. 
       Kids can learn with fun \u003cb\u003eEarth 
       Day\u003c/b\u003e activities, environmental projects, 
       \u003cb\u003e...\u003c/b\u003e"}],
     "cursor":{"pages":[
     {"start":"0","label":1},
     {"start":"4","label":2},
     {"start":"8","label":3},
     {"start":"12","label":4},
     {"start":"16","label":5},
     {"start":"20","label":6},
     {"start":"24","label":7},
     {"start":"28","label":8}],
      "estimatedResultCount":"40300000",
      "currentPageIndex":0,
      "moreResultsUrl":"http://www.google.com/search?
        oe\u003dutf8\u0026ie\u003dutf8\u0026source\u003duds\u0026start\
        u003d0\u0026hl\u003den\u0026q\u003dEarth+Day"}}, 
      "responseDetails": null, "responseStatus": 200}

This article demonstrates how to make such a call and use the response in a .NET project.

One of Google's requirements in their terms of use is that attribution to Google for the search results is made. The web controls in this project include JavaScript that print the 'powered by Google' logo on the results page.

The focus here is on standard web search results, although the API can return images and other types of search results. Modification of the code is required if other search result types are required.

More about interaction with the Google Search REST API can be read here: http://googlesystem.blogspot.com/2008/04/google-search-rest-api.html.

Background

The REST API does not have to be used server side, the results can equally be parsed client side using JavaScript. We, however, had a need to use the Google Search API to generate and record information about the search results returned, so a server side solution was what we came up with. An added advantage is that we have been able to build reusable .NET web controls that can be placed in any web project, with next to no configuration.

One of the advantages of using the Google Search REST API is that no API key is required and there is no limitation on the number of queries, making the code and controls here easy to reuse in new applications.

One disadvantage of the REST API is that it will only return results in groups of 8 per query, and there is only access to the top 32 results for any search term. The code here partly solves that problem by creating multiple queries and grouping results where more than 8 results are required, so that issue will not have to be dealt with by the developer implementing search functionality.

Google Search is a powerful tool, and integration server side has many practical applications, so we decided to contribute our code to help other .NET developers who want to make use of the REST API. The code published along with this article uses the API to create a simple web site search box, which redirects the user to a results page, although the modular structure of the solution is designed to allow for use of the code in other applications.

Screenshots

Integration on a live site can be viewed at: Cognize.co.uk.

Points of Interest

The Google Search REST API returns a JSON response. One of the more interesting bits of code in this article is the deserialisation of the JSON response to .NET objects. This required the construction of objects that represent the JSON response from the API. The objects were constructed following testing and observing the response data. The code makes the call using a WebClient. This is because the Google Search REST API requires a valid HTTP header, and this was the easiest way of achieving this.

The code then makes use of DataContractJsonSerializer to deserialize the response data into .NET objects:

/// <summary>
/// Get google search results for a particular site
/// </summary>
/// <param name="siteName">The fully qualified
/// site root path for the site that results are to be limited to.
/// E.g. http://www.cognize.co.uk</param>
/// <param name="searchString">The raw
/// (non encoded) search string</param>
/// <returns></returns>
public static ResponseData GetSearchResultsChunk( string siteName, 
       string searchString, int resultCountStartPos )
{
    ResponseData responseData = null;

    try
    {
        searchString = HttpUtility.UrlEncode( searchString.Trim() );

        using (WebClient client = new WebClient())
        {
            // Manipulate request headers - Google REST API
            // requires valid result header
            // hence the use of Web client as opposed to WebRequest
            client.Headers.Add( "user-agent", 
              "Mozilla/4.0 (compatible; MSIE 6.0; " + 
              "Windows NT 5.2; .NET CLR 1.0.3705;)" );

            string siteSearchString = String.Empty;

            if (!String.IsNullOrEmpty( siteName ))
            {
                // Param name and value must include no spaces
                siteSearchString = "site:" + siteName + " ";
            }

            // Result size is rsz in query string. This parameter causes the
            // api to return results in sets of 8 where 'large' is used
            // rather than 4 if 'small' is used,
            // so less total api requests are required.

            string resultSize = "large"; 

            string searchRequestURL = "http://ajax.googleapis.com/" + 
              "ajax/services/search/web?v=1.0&start=" + 
              resultCountStartPos.ToString() + "&rsz=" + 
              resultSize + "&q=" + siteSearchString + searchString;

            DataContractJsonSerializer jsonSerializer = 
              new DataContractJsonSerializer( typeof( GoogleAPISearchResults ) );

            // Read our search results into a .net object
            GoogleAPISearchResults searchResultsObj = 
              (GoogleAPISearchResults)jsonSerializer.ReadObject( 
              client.OpenRead( searchRequestURL ) );

            responseData = searchResultsObj.responseData;
        }
    }
    catch (Exception ex)
    {
        // Log error here

        // Allow exception to bubble up

        throw ex;
    }

    // Return response data including search results

    return responseData;
}

Objects representing the JSON response require the attribute [DataContract] at the class level, and [DataMember] for each property.

using System;
using System.Runtime.Serialization;

namespace Cognize.GoogleSearchAPIIntegration
{
    [DataContract]
    public class Pages
    {
        [DataMember]
        public int start
        {
            get;
            set;
        }

        [DataMember]
        public int label
        {
            get;
            set;
        }
    }
}

The management of multiple queries and grouping of results (due to restriction where the API will only return 8 results per call) is managed by the following code. If more than 8 results are required, additional calls are made using a different start index, so for example, if 10 results are requested, the second call will be made using 8 as the result start parameter (chunkResultCountStart).

One odd bit of behaviour in the Google Rest API is that if only a small number of results are returned for a search term, say for example 1, then a second call to the API for the second chunk will return the same search result, even if the start parameter is set to 8 - for the 8^th result. To get around this, on reading the first query result, the response data is read for the estimated result count (estimatedResultCount). If this value is lower than the maximum search results requested, then maxChunksRequired is modified within the for loop (not great practice I know, but necessary) to prevent excess calls to the API and duplicate results.

/// <summary>
/// Get google search results from a specific domain.
/// Google documentation suggests that the maximum number
/// of results that can be requested without
/// throwing an exception are 32. In testing, using large chunks,
/// upto 64 results have been acheived. Any number above 64 is amended to 64.
/// </summary>
/// <param name="siteName">The fully qualified site name url.</param>
/// <param name="searchString">The raw search string.</param>
/// <param name="requestedResults">The number of results requested. </param>
/// <returns>The number of search results to return.</returns>
public static SortedList<int, Results> 
      GetSearchResults( string siteName, 
      string searchString, int requestedResults )
{
    SortedList<int, Results> searchResultsLst = 
                                   new SortedList<int, Results>();

    // API will error if we request more than 64 results,
    // so modifiy if over that figure
    if (requestedResults > 64)
    {
        requestedResults = 64;
    }

    int maxChunksRequired = CalculateChunksRequired( requestedResults );

    for (int chunkIndx = 0; chunkIndx < maxChunksRequired; chunkIndx++)
    {
        // Results are returned in sets of 8
        // so we request results from the start of
        // the next group of 8.
        int chunkResultCountStart = (chunkIndx * 8);

        // Return the response data including chunk of search results
        ResponseData responseData = GetSearchResultsChunk( siteName, 
                     searchString, chunkResultCountStart );

        // For some search terms,
        // the max no of requested results will be higher
        // than the actual number of results google has.
        // This is determined
        // by esitmatedResultCount value
        // returned by the API. In this case we need
        // to reduce the number of API calls,
        // since superflous calls will result
        // in the results being duplicated
        // where google returns repeats the results despite
        // the value passed in chunkResultCountStart

        if (responseData != null)
        {
            if (chunkIndx == 0)
            {
                int realChunksRequired = CalculateChunksRequired( 
                         responseData.cursor.estimatedResultCount );

                if (maxChunksRequired > realChunksRequired)
                {
                    maxChunksRequired = realChunksRequired;
                }
            }

            // Put the results in a more manageable sorted list
            for (int resultIndx = 0; resultIndx < 
                        responseData.results.Length; resultIndx++)
            {
                searchResultsLst.Add( searchResultsLst.Count, 
                        responseData.results[resultIndx] );
            }
        }
    }

    return searchResultsLst;
}

Using the Code

To use the code, you can use the web controls included if you just want to plug it straight into your web app.

Should you want to use the code for other purposes, the key method to use is:

public static SortedList<int, Results> 
       GetSearchResults( string siteName, 
       string searchString, int requestedResults )

As it says 'on the tin', this will retrieve search results for a given search term up to the number of results requested (maximum in testing was 64).

An overload that does not restrict results to a particular site is available, that can be used in a similar way.

History

• 30^th December 2009: First version published.