7 Easy Steps to Building Your Idea Around Amazon Web Services

• Download source files - 46.7 KB

Introduction

This article will use code to demonstrate how easy it is to build an application on Amazon S3. We’ll also highlight questions that you may ask before releasing an application. For illustration purposes, the text in italics represents the thought process of a developer (in this case, me) who is building an application (in this case, S3Whois).

Step 1: Start with Your Cool Idea

Every business starts with a cool idea. Either you already have one, or you have a problem that needs a solution. You start small, but think big. Just ask a few simple questions to test your idea:

• Has anybody done it yet? Is my idea unique?
• Is there a potential to earn money (business model)?
• What problem does it solve?

Believe in yourself and your idea, and always validate your idea with "been-there-done-that" folks.

I came up with a simple idea. I thought it would be cool to show a small tooltip box when you move your mouse over any e-mail address that appears on any web page while you are surfing. We can be very creative about what to show in the tooltip. To start off, I thought I will simply display all the "business card" information of the individual with a small photo. Something like this:

I thought the idea is cool because it gives e-mail a face, and helps me associate an e-mail address with an actual identity/person.

Step 2: Whiteboard Your Idea

How would the overall system work? Always use the whiteboard when you need to dump your brain. There will be lots of "idea threads" spawning, which will, in turn, spawn more "idea threads" and not let you focus on the main idea. With the goal to get something fast, it's important that you stay focused on your main success scenario. Whiteboards usually help transcribe your thoughts. Questions that you need to ask are:

• How would the overall system work?
• What are the main pieces of the puzzle? Does it have a Web Server that serves your web pages? Does it have a client?
• How will the different pieces interact with each other?
• How will the system look and feel?

My idea of trying to make it work for every web page on the Internet suggests that I need to develop a plug-in or toolbar for browsers. The plug-in will parse each page the user visits and replaces all the mailto: href links with injected JavaScript code. This will render a small div layer (size of business card) when the mouse-over event is fired.

The tooltip will then query a Web Server and data store using the e-mail as an input parameter, then fetch me data associated with the e-mail. Initially, the data can be an image (photo) and some additional information like name, e-mail address, interests, and zip code.

How would I get the data in the first place? I would need a web form that users can use to insert data into the data store.

Initially, my whiteboard diagram looked like:

My main pieces of the puzzle were:

1. Browser plug-in/toolbar/script will serve as the client.
2. Web Server will have two pages. One fetches the data from the data store, and the other inserts the data into the data store via an HTML form.
3. Need some data store that will store my end-user data.

How will the system look and feel?

• The Get page needs to look like a business card with a photo on the side
• The Put form needs an HTML form. I could have embedded the form using the free website templates (or Microsoft VS.NET 2005 templates)

Overall, I felt I needed the following:

• Intuitive UI: Tooltip should be available when the user visits any web page on the Internet.
• Scalable, reliable, fast storage: I don't have enough money to spend. Storage should be economical. Storage retrieval should be reliable and fast.
• Fast development: Something that has a small learning curve.

With the overall requirements of:

• Reuse: Reuse as much as possible. Don't re-invent the wheel.
• Easy to develop: Development and integration should be easy. I don't want to get stuck in database SQL query writing and lose the focus of my actual idea.

Step 3: Get the Right Tools

So you have a cool idea. And you also have a good feel of how the overall system will work and how the different parts will interact with each other. Now, we want to delve a little deeper and ask the following questions:

1. Which technology platform should I use?
2. Which Open Source / third-party utility tools can I use?
3. What if I get famous? How big will it scale?
4. Where will I host my application?

It's important to get there fast, and hence it's important that you have the right equipment/gear to get there. Either people decide to choose something they are already familiar with, or they take this opportunity to learn a new product or technology.

I chose to do the latter. Taking the above pieces, and with an aim to develop something fast (in one weekend), I thought of the following:

To address #1 above, I thought it would be cool to use something like GreaseMonkey and install a small user script (probably might already be developed by some genius developer and just waiting for me to get noticed - think positive) in the script repository. By this, users (my customers) just have to install one cool little script that will do all the magic. One caveat: Greasemonkey just works with Firefox, but there are also similar engines like greasemonkIE and turnabout that works for IE too. Other alternatives were: developing a BHO/ActiveX IE plug-in or Firefox plug-in.

To address #2, I decided to use VS.NET 2005 that comes with a built-in IIS server. I thought it would be a cool opportunity for me to learn the new product from Microsoft.

To address #3, I thought the fastest and easiest way to develop will be to consume an already existing Web Service. Instead of creating databases and writing the same-old CRUD database SQL logic, I will simply use some sort of Web Service that will store all the profiles of my customers.

Hence, I decided to use the new Amazon S3 Web Service for three main reasons:

• It's reliable.
• It was dirt cheap (15c/GB to store data, and 20c/GB-month-transfer), and
• It's scalable - I don't have to worry about my app getting famous overnight (aha!). I don't have to worry about infrastructure because it all comes from Amazon and as a pay-as-I-go service. It's using a pay-per-use metaphor - just like my on-demand direct TV service where I only pay for the movies that I like to see.

Sweet, isn't it?

A New Whiteboard Diagram

Got the Right Tools/Code-Generators

I installed the Greasemonkey plug-in for Firefox, and modified a user script that was available from their user script repository (credits to Walter Zorn for providing his cool JavaScript tooltip). I already had VS2005.NET and IIS configured and ready.

Created an AWS Account

I went to http://aws.amazon.com, created an account, and got the Access Key ID and the Secret Access Key. I also signed up for Amazon S3 by clicking on the "Sign up for Web Service" button on http://aws.amazon.com/s3 and providing my credit card details.

Downloaded Helper classes

• Amazon has provided starter kits in a variety of languages like Java and PHP for both REST and SOAP interfaces. I got the C# REST and SOAP APIs from the Amazon S3 Code Samples.
• Read the Technical Documentation at the Resource Center.

Step 4: Start Coding!

It is strongly recommended to dive into the code. It's similar to the approach that we take when we learn how to swim. Jump! You will flip-flop-splish-splash and strive to survive for a few minutes; however, after some time, you will feel all your phobia has transformed into confidence. Likewise, you will realize how easy it is to integrate Amazon S3 in your application only if you take the plunge into the code. Questions that you should ask here are:

• Where would I start from? Which is the best starter pack that can get me closer to what I need?
• Is there a code sample already in the AWS Resource Center that match my needs? Or do I need to build one prototype from scratch?
• How can I make a call to a Web Service? What bootstrap code do I need?

I created a new web site in VS.NET 2005 and added existing files that I downloaded off the Resource Center in the code sample sections. I had all the code that I needed to create my first Happy-path Use Case. My Solution Explorer looked something like this:

I created a new file called getprofile.aspx and inserted the following code:

<asp:Panel ID="profilePanel" runat="server">
  <table>
     <tr>
      <td rowspan="4">
          <form action="" runat="server">
             <asp:Image ID="s3WhoisImage" runat="server" Height="150px" Width="100px"/>
          </form>
      </td>
      <td>
       <table>
         <tr> <td>
          <asp:Label ID="s3WhoisName" runat="server" Text="Label"></asp:Label></td>
      </tr>
      <tr><td>
          <asp:Label ID="s3WhoisEmail" runat="server" Text="Label"></asp:Label></td>
      </tr>
       <tr><td>
          <asp:Label ID="s3WhoisInterests" runat="server" Text="Label"></asp:Label></td>
      </tr>
       </table>
      </td>
      </tr>
  </table>
</asp:Panel>

To back this HTML, I had the following in my code (getprofile.aspx.cs):

//amazon aws credentials : you can get these from creating one at http://aws.amazon.com
//I am reading from web.config
string accessKey = system.Configuration.ConfigurationManager.AppSettings.Get("AccessKey");
string secretKey = System.Configuration.ConfigurationManager.AppSettings.Get("SecretKey");
  
//use your bucketname from web.config
bucketName = System.Configuration.ConfigurationManager.AppSettings.Get("bucketname");

//using REST get the URL of the image. Let the browser make the fetch call
//tricky to do this in SOAP as ASP.NET does not have 
//ImageControl that takes in byte Array/bitmap
//QueryStringAuthGenerator simply generates
//the URL that we can use to make the REST call
QueryStringAuthGenerator generator = 
   new QueryStringAuthGenerator(accessKey, secretKey, true);
generator.ExpiresIn = 60 * 1000;
string url = generator.get(bucketName, s3WhoisId, null);
s3WhoisImage.ImageUrl = url;

// *******************************************************
//  REST example to get metadata of S3 object
// *******************************************************
RESTConnection restS3 = new RESTConnection(accessKey, secretKey);
GetResponse resp = restS3.getHead(bucketName, s3WhoisId, null);
SortedList metadata =  resp.Object.Metadata;

//populating the html fields from the metadata
s3WhoisEmail.Text = (string)metadata["s3WhoisEmail".ToLower()];
s3WhoisName.Text = (string)metadata["s3WhoisName".ToLower()];                
s3WhoisInterests.Text = (string)metadata["s3WhoisInterests".ToLower()];

I found that I could also do the same thing using SOAP:

// *******************************************************
//  SOAP example to get metadata of S3 object
// *******************************************************
SOAPConnection s3 = new SOAPConnection(accessKey, secretKey);
GetObjectResult objResult = s3.get(bucketName, s3WhoisId, true);
s3WhoisEmail.Text = GetMetadataValue(objResult.Metadata, "s3WhoisEmail");
s3WhoisName.Text = GetMetadataValue(objResult.Metadata, "s3WhoisName");
s3WhoisInterests.Text = GetMetadataValue(objResult.Metadata, "s3WhoisInterests");

GetProfile.aspx looked something like this:

OK. So now I learned the way go "GET" an object from S3 using SOAP and REST. I needed to find out how I can "PUT" an object in Amazon S3. The PUT object was slightly tricky as I found out that there are multiple design choices:

1. I could have an image stored in Amazon S3 as an object with an "image" content-type, and "attach" all the other profile data as meta-data of that object.
2. I could have an image stored in Amazon S3 as an object with an "image" content-type, and have profile data also as objects of text content-type.
3. I could have an image stored in Amazon S3 as an object, with an additional text file object that stores all my profile data.

As you can see, Amazon S3 gave me the flexibility to choose what's best for me. #1 seems to be the most viable option in my case as I could get the object and all its metadata in one single request, and thereby it reduces the number of requests. Plus, it's more scalable as I just have to add a line of code or two to add a new metadata object.

This is what my putprofile.aspx looks like:

My code-behind putprofile.aspx.cs:

public partial class PutProfile : System.Web.UI.Page
{
    private string bucketName = "";
    private SOAPConnection s3;
    private RESTConnection restS3;
    protected void Page_Load(object sender, EventArgs e)
    {
        //amazon aws credentials : you can get these
        //from creating one at http://aws.amazon.com
        string accessKey =
          System.Configuration.ConfigurationManager.AppSettings.Get("AccessKey");
        string secretKey =
          System.Configuration.ConfigurationManager.AppSettings.Get("SecretKey");
        //use your bucketname
        bucketName = 
          System.Configuration.ConfigurationManager.AppSettings.Get("bucketname");
        //we will demostrate both SOAP and REST. so lets instantiate the connection
        s3 = new SOAPConnection(accessKey, secretKey);
        restS3 = new RESTConnection(accessKey, secretKey, true);
        string id = Request.QueryString["id"];
        if (id != "" && id != null)
        {
            s3WhoisId.Enabled = false;
            s3WhoisId.Text = id;
        }
        else
            s3WhoisId.Enabled = true;
    }
    protected void cmdSend_Click(object sender, EventArgs e)
    {
        // Check to see if image file was uploaded
        if (filMyFile.PostedFile != null)
        {
            // Get a reference to PostedFile object
            HttpPostedFile myFile = filMyFile.PostedFile;
            // Get size of uploaded file
            int nFileLen = myFile.ContentLength;
            // make sure the size of the file is > 0}
            if (nFileLen > 0)
            {
                // Allocate a buffer for reading of the file
                byte[] myData = new byte[nFileLen];
                // Read uploaded file from the Stream
                myFile.InputStream.Read(myData, 0, nFileLen);
                // Create a name for the file to store
                string strFilename = Path.GetFileName(myFile.FileName);
              // *******************************************************
              // using REST to put a file in S3
              // *******************************************************
                SortedList metadata = new SortedList();
                metadata.Add("s3WhoisName", s3WhoisName.Text);
                metadata.Add("s3WhoisEmail", s3WhoisEmail.Text);
                metadata.Add("s3WhoisInterests", s3WhoisInterests.Text);
                SortedList headers = new SortedList();
                headers.Add("Content-Type", "image/jpeg");
                //prepare the object
                S3Object obj = new S3Object(myData, metadata);
                //make the call
                Response res = restS3.put(bucketName, s3WhoisId.Text, obj, null);
                //check the response message
                if (res.Status == System.Net.HttpStatusCode.OK)
                message.Text = "Profile Stored";
            }
        }
    }

I also found that I could do the same using SOAP:

// *******************************************************
// using SOAP to put a file in S3
// *******************************************************
PutObjectResult objS3PutObjectResult;
MetadataEntry objMetaDataEntry;
MetadataEntry[] aobjMetaDataEntries = new MetadataEntry[3];

// adding some metadata entries to the object data
objMetaDataEntry = new MetadataEntry();
objMetaDataEntry.Name = "s3WhoisName";
objMetaDataEntry.Value = s3WhoisName.Text;
aobjMetaDataEntries[0] = objMetaDataEntry;
objMetaDataEntry = new MetadataEntry();
objMetaDataEntry.Name = "s3WhoisEmail";
objMetaDataEntry.Value = s3WhoisEmail.Text;
aobjMetaDataEntries[1] = objMetaDataEntry;
objMetaDataEntry = new MetadataEntry();
objMetaDataEntry.Name = "s3WhoisInterests";
objMetaDataEntry.Value = s3WhoisInterests.Text;
aobjMetaDataEntries[2] = objMetaDataEntry;

// make the remote call
objS3PutObjectResult = s3.put(bucketName, s3WhoisId.Text, 
                       myData, aobjMetaDataEntries, null);
if (objS3PutObjectResult != null)
message.Text = "Profile Stored";

This is how it looks in the end:

In summary, this application utilizes the following things:

• Visual Studio 2005 IDE (.NET 2.0 framework)
• Amazon S3 C# library
• Greasemonkey engine (free)
• Greasemonkey user script

GetProfile.aspx gets the stored image/profile and its metadata from S3 using a unique identifier. The unique identifier is stored as a "key" of the S3 object in S3. PutProfile.aspx is an ASP.NET web form that takes information like image, name, email address from a web page and stores the information in the form of an object in S3. All images are stored as "objects" in the 'S3whois' bucket, and additional information is stored as metadata of that object. The Greasemonkey script parses the HTML page and injects the mouse-over JavaScript code wherever mailto: links are encountered. Mouse-over displays the getprofile.aspx page.

Step 5: Release Your "Happy Path" Beta

There are always edge cases/exceptions in any application. Some tips that might be helpful:

• Note all your edge cases.
• Analyze the implemented use cases to see whether they open up any security holes.
• Safe guard your AWS credentials. Make sure when you distribute and package your application, you clear those fields.
• Test your application by sending it to your colleagues and friends/family, and of course, send it to even Amazon Web Services. They always would like to know what you are doing.

As soon as my "Happy Path" was working, I hosted my app on my local Web Server, so that other people could start using it and provide me with more feedback. I documented the main success scenario and created a one page "user manual".

Step 6: Publicize / Blog / Document and Let the World Know

It is important to share your application with others. Let the world know about you. Let the world blog about you.

I did a number of things to market and publicize my app:

• AWS' Resource Center always has cool stuff. I found the "Solutions Catalog" to be a popular place where all the developers list their app. I filled out a simple form, and added info about my application in the solutions catalog. I also found out what other developers were building with Amazon S3.
• I used the Developer Forums to ask my questions.
• I also sent an email to awseditor@amazon.com and informed them about my app.
• I blogged about it on my personal blog site.

Step 7: Keep Improving and Adding New Features to Stay Ahead in the Game

Continuously keep track of features so that you can add and improve. One of the ways that has proven successful is to maintain a wiki page where others can add content (features/comments) about your application.

I chose to create an Excel sheet that helps me track/sort the status of each new feature that I add. I continuously maintain and update that Excel worksheet whenever any new interesting feature crops up.

Resources

• Amazon Web Services website
• Amazon S3 Resources