Click here to Skip to main content
Click here to Skip to main content

Goodreads API Tutorial

, 7 Oct 2012 CPOL
Rate this:
Please Sign up or sign in to vote.
How to query an API using Goodreads.com as an example.

Introduction 

In this article we'll go through the process of setting up a web page that can query information using another site's API. As an example, we'll use the API provided by Goodreads, and our goal will be to display general information about a book when we provide an author and book title. In order to accomplish this, we'll use Javascript with a little jQuery, an HTTP Handler with C# as the back end, and discuss converting XML to JSON in order for our Javascript to quickly read the data we query. Finally, we'll implement how to cache the data in order to reduce server load.

Before We Begin

Typically, in order to use an API, you'll need a developer key. This is a unique key tied to your account. It can be banned if abused, so it is important to read the terms of use associated with an API before you implement it on your site. To get a developer key, and to see the different methods available, for Goodreads, visit http://www.goodreads.com/api. The particular method I'll be calling in this article is book.title. Once you have your developer key, a sample URL is provided where you can see an example of the XML that we'll get back, and the information therein. Notice the formatting of the URL; it will be the pattern we'll need to follow as well.

We'll also be using a little jQuery, so we'll need to download the jQuery Javascript file; it can be found at http://jquery.com/download/. I recommend the minified version for this exercise. Once you download the file, you can drag and drop it into your solution window; though, I recommend putting it in its own folder inside your solution called "scripts".

Creating a Client Control

First, let's create a simple test page to display our data. All that I will be putting on this page is some basic HTML and a call to a Javascript function that will display the data.

<body>
    <form id="form1" runat="server">
    <h2>Get Book Information</h2>
    Author: <input type="text" id="authorTextbox" value="Patrick Rothfuss" /> <br />
    Title: <input type="text" id="titleTextbox" value="The Name of the Wind" /> <br />
    <input type="button" value="Get Book Information" id="getButton" onclick='getBookInformation()' />
    <br /><br />
    <div id="DataContainer" ></div>
    </form>
</body>

I've added a text box for the author and a text box for the book title. I've also added values for the text boxes already, so that we can easily test if the page is working without constantly adding in values. You certainly don't need to do that though. In addition to the text boxes is a button, that when clicked, calls a Javascript function called getBookInformation. We'll define this in a separate Javascript file later.

The final important item is the div. It's blank now, but in our js file we'll reference that div and populate it with the data we get from Goodreads. It must have an id that we can reference, and here, I'm just calling it DataContainer.

We'll need to add a reference on this page to our js file once it's created, otherwise we're done with this page, and we'll let the Javascript take care of the rest.

Creating the HTTP Handler and Javascript 

The HTTP Handler is where we'll be specifying where to get our data, and how to format it once we've got it. First, we'll add a Generic Handler to our project. I've called mine goodreadsHttpHandler.ashx. We'll write our code inside the ProcessRequest method, and we'll leave IsReuseable alone. Before we forget, we want to change the Response.ContentType. We're going to be getting XML from Goodreads, then converting it to JSON, so we want to change the content type to "application/json" to reflect what we're ultimately returning.

context.Response.ContentType = "application/json";

Next, we're going to add a Javascript file to our solution. For now, we'll just add a simple function that gets the information in our text boxes and passes it to the HTTP Handler.

function getBookInformation() {
	$.get('goodreadsHttpHandler.ashx'
	, { bookAuthor: $('#authorTextbox').val(), bookTitle: $('#titleTextbox').val() }
	, function (data) {
	}
	);
}

Essentially, what's happening here is the function is looking up the handler and then passing it the information it needs (in this case the bookAuthor and bookTitle). Then, we pass into the function, in this case what we're calling "data", what the handler returned. Later, we'll display the data as HTML which we define inside the function. Note that the authorTextbox and titleTextbox are the id's of the text boxes we created on the client control page.

Back in our handler, we need a way to reference what the Javascript passes into it. We do this using the QueryString method.

string bookTitle = context.Request.QueryString["bookTitle"];
string bookAuthor = context.Request.QueryString["bookAuthor"];

Now that our handler knows the book title and book author, it has all the information it needs to query the Goodreads API. So let's create a new method in our handler called GetGoodreadsURI, and it'll take the bookAuthor and bookTitle as parameters.

public string GetGoodreadsURI(string bookAuthor, string bookTitle)
{
    string myKey = keyManager.GetConfigurationByKey("goodreadsDeveloperKey");
    string uri = "http://www.goodreads.com/book/title?format={0}&author={1}&key={2}&title={3}";
    return String.Format(uri, "xml", bookAuthor, myKey, bookTitle);
}

In my example, I'm getting my developer key from another class that refers to the web.config file. Your key should also be hidden from your users in a similar fashion. To add the file to your web.config, add the following into the "configuration block".

<appSettings>
    <add key="goodreadsDeveloperKey" value="YOUR_KEY_HERE"/>
</appSettings>

Whether you add it to another class or not, in order to reference this inside your web.config, you would use:

WebConfigurationManager.AppSettings["goodreadsDeveloperKey"];

Returning to our GetGoodreadsURI method, notice that for the format we're choosing XML. This particular API Method that we're calling allows us to choose the format as JSON specifically. However, if we do, Goodreads will only return reviews. Since we'd like to get more information about a book, we'll make the call using XML (which provides more general information) and convert it to JSON.

Back in our ProcessRequest method, we'll call our new GetGoodreadsURI method, and call its return value uri.

string bookTitle = context.Request.QueryString["bookTitle"];
string bookAuthor = context.Request.QueryString["bookAuthor"];
string uri = GetGoodreadsURI(bookAuthor, bookTitle); 

Now that we know where to look for the XML, we want to copy it into a string that we can then turn into JSON.

public string GetResponseFromAPI(string uri, out int serverStatus)
{
    string responseData = String.Empty;
    try
    {
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
        HttpWebResponse res = (HttpWebResponse)req.GetResponse();
        serverStatus = (int)res.StatusCode;

        using (Stream s = res.GetResponseStream())
        {
            using (StreamReader sr = new StreamReader(s))
            {
                responseData = sr.ReadToEnd();
            }
        }
        return responseData;
    }
    catch (WebException e)
    {
        HttpWebResponse res = (HttpWebResponse)e.Response;
        serverStatus = (int)res.StatusCode;
        return responseData;
    }
}

First, we're simply making the request and then storing the response. After that, we're reading through the entire response that's been returned. Finally, we put it into a string that we can easily read from later.

You don't need to grab the StatusCode the server returns, but for debugging purposes, or to supply relevant error messages to your users, it might be valuable. If everything is successful, we'll get a StatusCode of 200. In the above code, the only exception I'm catching is if we get a 404 not found (i.e. the user searches for a book that Goodreads can't find in its database).

Now that we have the information from Goodreads captured into a string, we can convert it to JSON to send back to our Javascript function. Converting it isn't difficult, but it is tedious. To start with, we'll want to create a text file that maintains the JSON format we expect to send to our Javascript. You can look through the sample XML file Goodreads provides in order to determine the information you want to return back to the user. Based on what I've chosen, this is what my text file looks like:

{
	"Author":  "{{!Author!}}"
	, "Title":  "{{!Title!}}"
	, "Description": "{{!Description!}}"
	, "Average_Rating": "{{!Average_Rating!}} / 5"
	, "Cover_Image": "{{!Cover_Image!}}"
	, "Publication_Year": "{{!Publication_Year!}}"
	, "Publisher": "{{!Publisher!}}"
	, "ISBN": "{{!ISBN!}}"
	, "Reviews": "{{!Reviews!}}"
	, "Status": "{{!Status!}}"
} 

Remember our empty Javascript function from before? If we passed the content of this text file to it, and called it data (which we are), we could then reference data.Author, and we would get back {{!Author!}}. What we're working toward is taking information from the string that holds our XML information, and replacing the relevant fields in our text file (which we'll put into a string also). So by replacing {{!Author!}} with, say, Patrick Rothfuss, we'll get back that author name when we call data.Author in our Javascript function.

Now that we have this text file, let's get it into a string. I'm going to create a new method called GetGoodreadsJSONResponse.

public string GetGoodreadsJSONResponse()
{
    string returnData;
    using (Stream s = Assembly.GetExecutingAssembly().GetManifestResourceStream(
	"YOUR_SOLUTION_NAME.JSON_Responses.goodreads.BookInformationFields.txt"))
    {
        using (StreamReader sr = new StreamReader(s))
        {
            returnData = sr.ReadToEnd();
        }
    }
    return returnData;
}

In my example, I have a folder called “goodreads” inside a folder called “JSON Responses”. Within the “goodreads” folder is my text file. You don't have to follow that folder structure, but do note that with the GetManifestResourceStream method, you pass a file name using periods where you'd normally see slashes. Otherwise, all this method does is return our text file as a string.

Now, we can add these two new methods to our ProcessRequest.

string returnData = GetGoodreadsJSONResponse();
string responseData = GetResponseFromAPI(uri, out serverStatusCode);
if (serverStatusCode == 200)
{
	returnData = ConvertGoodreadsXMLtoJSON(responseData, returnData);
}

In the code above, there's one method that we haven't yet defined: taking everything in the response string (formatted as xml) and putting it into our return string (formatted as JSON). Unfortunately, converting the XML to JSON is a large method:

public string ConvertGoodreadsXMLtoJSON(string responseData, string bookFieldsAsJSON)
{
	XDocument xmlResponseData = XDocument.Parse(responseData);
	XElement goodreadsRoot = xmlResponseData.Element("GoodreadsResponse");
	XElement bookRootElement = goodreadsRoot.Element("book");
	XElement workRootElement = bookRootElement.Element("work");
}

The first thing we're doing is creating an XML document based on the string that has the response data. Our string is already formatted as XML, so this is no problem. Next, we need to dive into each element to get information out of it. So if you look at the sample XML page again, you'll notice that there's an element called "GoodreadsResponse" that has all of the other elements within it. Inside that element is another element called "book" that houses everything related to the actual book. Under the "book" element is an element called "title". Now you'll notice that the "title" element actually has information we're looking for (particularly the name of the book).

XElement titleElement = bookRootElement.Element("title");
string bookTitleValue = titleElement.Value;

So we've essentially drilled down into the "title" element, and then gotten the value inside of it. To get the name of the author, you'd have to go from "GoodreadsResponse" to "book" to "authors" to "author" to "name" before you'd be at the element that has the value you want. Thus, it can be a tedious process to do for every single piece of information.

Now that we have the value in the title element, we want to replace it with the placeholder value in our JSON string.

bookFieldsAsJSON = bookFieldsAsJSON.Replace("{{!Title!}}", bookTitleValue);

You would now go through this process for each value you want to extract from the original XML. However, one problem is that the "reviews_widget" and the "description" values can contain HTML. If you put this HTML as is into your JSON string, it will not work. For this, we have to serialize it (i.e. convert it to a JSON-ready string).

JavaScriptSerializer js = new JavaScriptSerializer();
XElement descriptionElement = bookRootElement.Element("description");
string bookDescriptionValue = js.Serialize(descriptionElement.Value).Trim();
bookDescriptionValue = bookDescriptionValue.Substring(1, bookDescriptionValue.Length - 2); 

The last two lines of code might seem strange, but they're necessary every time we use the Serialize method. Serialize will automatically put quotes around the string it creates, but as you'll recall, in our JSON text file, we already have quotes around each value that we want to display. The last two lines of code remove the superfluous quotes that get created around the string. Without getting rid of them, it will not work.

You'll need to serialize anything that might have HTML in it, so you'll definitely need to do this with the "reviews_widget" if you want to display it. You do NOT need to serialize URLs though, so you can pass the value for "image_url" as is without modifying it.

In case a book can't be found, I want to display an appropriate message, so I have a status field in my JSON text file. At the end of converting all the XML to JSON, I change the status to OK and then return my new string.

bookFieldsAsJSON = bookFieldsAsJSON.Replace("{{!Status!}}", "Status_OK");
return bookFieldsAsJSON;

Now we're ready to send this back to our Javascript function. So at this point, our ProcessRequest method is complete and functional, though later we'll come back and add caching.

public void ProcessRequest(HttpContext context)
{
	context.Response.ContentType = "application/json";
	
	string bookTitle = context.Request.QueryString["bookTitle"];
	string bookAuthor = context.Request.QueryString["bookAuthor"];
	
	string uri = GetGoodreadsURI(bookAuthor, bookTitle);
	
	string returnData = GetGoodreadsJSONResponse();
	string responseData = GetResponseFromAPI(uri, out serverStatusCode);
	if (serverStatusCode == 200)
	{
		returnData = ConvertGoodreadsXMLtoJSON(responseData, returnData);
	}
	context.Response.Write(returnData);
}

Completing the Javascript Function 

Finally, our getBookInformation function is ready to do some work. We're going to use jQuery to insert HTML onto our page that contains the information our handler returned. To do this, we'll add a simple jQuery command inside our function.

function getBookInformation() {
	$.get('goodreadsHttpHandler.ashx'
	, { bookAuthor: $('#authorTextbox').val(), bookTitle: $('#titleTextbox').val() }
	, function (data) {
	$('#DataContainer').html(
	)
	}
	);
}

Remember, DataContainer is the name of the div we specified in our main HTML page earlier, so everything we put inside the .html() will get inserted into that div as HTML. This means that we can put HTML directly into that function, but everything that isn't Javascript has to be put inside quotation marks (including the HTML). You can style the content exactly how you want to using HTML and CSS, and it will automatically get filled into the div. To reference anything returned by our handler, we simply use data.whatever. So, if we want to get the author, we'd call data.Author (remember, these are based on the names you've given the values in your JSON text file).

function getBookInformation() {
    $.get('goodreadsHttpHandler.ashx'
    // Specify location of Author and Title to search for.
    , { bookAuthor: $('#authorTextbox').val(), bookTitle: $('#titleTextbox').val() }
    , function (data) {
    var titleStyle = "<p style=\"color:#666600;font-family:georgia,serif;\">";
    var spanStart = "<span style=\"color:black;\">";
    var spanEnd = "</span>";
    var reviewsDiv = "<div id=\"ReviewContainer\"><p style=\"cursor:pointer;color:#666600;font-family:georgia,serif;\">";
    var infoFromGoodreads = "<br/><br/><br/><p style=\"font-size:9px;\">
	Information provided by <a href=\"http://www.goodreads.com\">goodreads</a>.</p>";
    if (data.Status === "Status_OK") {
    // Specify div to fill (change in else statements below also).
    $('#DataContainer').html(
    "<table><tbody><tr>"
    + "<td style=\"width:100px;\">" + "<img src=" + data.Cover_Image + "></>" + "</td>"
    + "<td>" + titleStyle
    + "Author: " + spanStart + data.Author + spanEnd
    + "<br/>" + "Title: " + spanStart + data.Title + spanEnd
    + "<br/>" + "Average Rating: " + spanStart + data.Average_Rating + spanEnd
    + "<br/>" + "First Published: " + spanStart + data.Publication_Year + spanEnd
    + "<br/>" + "Publisher: " + spanStart + data.Publisher + spanEnd
    + "<br/>" + "ISBN: " + spanStart + data.ISBN
    + "</p></td></tr></tbody></table>"
    + titleStyle + "Description" + "<br/><br/>" + spanStart + data.Description + spanEnd + "</p>"
    + reviewsDiv + "Show reviews..." + "</p></div>"
    + infoFromGoodreads
    );
    } else if (data.Status === "Bad_XML") {
        $('#DataContainer').html("Unexpected XML encountered." + infoFromGoodreads);
    } else {
        $('#DataContainer').html("No book found." + infoFromGoodreads);
    }
    $("#ReviewContainer").click(function () { $('#ReviewContainer').html(data.Reviews); });
    }
    );
}

The last thing we need to do is add a reference to our script on the HTML page that's going to display this information. So go back to the client control and drag-and-drop both the Javascript we made, as well as the jQuery script, into the top of the HTML page.

<script src="scripts/jquery-1.8.0.min.js" type="text/javascript"></script>
<script src="scripts/goodreadsGetBookInfoJavascript.js" type="text/javascript"></script> 

At this point, we have a working webpage that calls the Goodreads API and displays the data we've requested. However, there's nothing stopping a user from tapping the "Get Book Information" button as fast as they can, and getting our developer key banned. Additionally, it generally isn't worthwhile to make a call to Goodreads over and over again if people are searching for the same information repeatedly.

Caching the Data 

First, let's create a new interface and call it ICacheData.

interface ICacheData
{
    string GetCacheValue(string key);
    void InsertToCache(string key, string value);
}

Now, create a new class called CacheManager that inherits from the interface we just made. This class only requires two methods. The first is to check if we already have something in the cache, and then return what's found. If something gets returned, we'll pass that to our user, rather than hitting Goodreads again.

public string GetCacheValue(string key)
{
    string returnData = null;
    if (null != HttpContext.Current.Cache[key])
    {
        returnData = HttpContext.Current.Cache[key].ToString();
    }
    return returnData;
}

Notice that we're accepting a string as a parameter. This needs to be something unique to each search. One possible key is the URI that we query to get the information to begin with (it contains the author and book title), so in our handler we'll pass that in as the "key" value.

Basically, we are just checking if that particular uri key is in our cache. If it is, we return the value that's stored (our JSON string), and if not, we just return null. Back in our handler, if we get null, we want to insert it into the cache.

public void InsertToCache(string key, string value)
{
    lock (_lockObject)
    {
        if (null == HttpContext.Current.Cache.Get(key))
        {
            HttpContext.Current.Cache.Add(
                key
                , value
                , null
                , DateTime.Now.Add(TimeSpan.FromSeconds(300))
                , System.Web.Caching.Cache.NoSlidingExpiration
                , System.Web.Caching.CacheItemPriority.Low
                , null
                );
        }
    }
}

Again, we're passing our URI as the key parameter. The value parameter will be the information we want cached. In our case, it's the final JSON string that we want to pass to our Javascript function. The other code to note is the DateTime.Now.Add(). What we're saying is to add 5 minutes to the current time (in production code, you would want to make this configurable), and that's how long our data will stay cached. If you expect that your data is going to be very dynamic and changing frequently, you may want to set that to something lower. On the other hand, if you expect your data won't change much, you could set this to a much longer span of time.

Now that we have methods to cache data, let's make our handler implement them.

CacheManager cm = new CacheManager();
string returnData = cm.GetCacheValue(uri);

if (null == returnData)
{
    returnData = GetGoodreadsJSONResponse();
    string responseData = GetResponseFromAPI(uri, out serverStatusCode);
    if (serverStatusCode == 200)
    {
        returnData = ConvertGoodreadsXMLtoJSON(responseData, returnData);
    }
    cm.InsertToCache(uri, returnData);
}

context.Response.Write(returnData);

As you can see, it's a fairly small addition to implement a simple form of caching data. We first check to see if data is cached, and if so, we just return it. If no data is cached, we go through the same code as before, but we cache it before moving forward.

*Note: You might notice that if you search for an author and capitalize the name, that data will get cached as expected. However, if you then search again but don't capitalize anything, it will cache the same data again for the different case. You could avoid this by putting all search terms to lower case before putting them in your URI. For whatever reason, at the time of this writing, if you put an author name in all lower case or all upper case, Goodreads will return a 404 error. If you're working with a different API, this hopefully won't be an issue.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Jesse G. Winston
Software Developer
United States United States
I am a developer in the Atlanta area with a focus on the Microsoft stack. My educational background is in math, and I apply those problem-solving techniques to the projects I develop.

Comments and Discussions

 
QuestionCannot get this to work--suggestions PinmemberMember 1042270324-Nov-13 11:20 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.141216.1 | Last Updated 7 Oct 2012
Article Copyright 2012 by Jesse G. Winston
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid