Click here to Skip to main content
Click here to Skip to main content

Exploring Amazon S3 with F#

, 15 Jul 2010 CPOL
Rate this:
Please Sign up or sign in to vote.
The article demonstrates how to explore the Amazon S3 API using F# and achieve great results just in 45 minutes.

Introduction

Traditional imperative languages teach developers that program code requires ceremony. You can't just write a line and expect it to do something. When programming book writers explain how easy it is to display the "Hello world" string in their favorite language, they actually start with an explanation that every call should be wrapped in a method, and the method typically takes some arguments etc. And of course, they have to let beginners know how to compile the resulting program and how to execute it. Without all these steps, the world won't get its greeting.

All these steps have valid reasons, but as a result, development environment based on major imperative languages do not really encourage exploration. It's best suited towards large scale development, but if a developer wants to have a quick look on something new, and he can't proceed without creating a new project, then he will often postpone this work until he has enough time or enough reasons to go ahead.

Luckily, there have always been alternatives to languages like C# or Java that worked better for rapid prototyping or technology exploration. One of the late additions is F#, and when I recently introduced myself to Amazon Simple Storage Service, I realized that if I need to get a quick overview of its programming facilities, using F# would be much more efficient than writing a test program in C#.

The purpose of this article is to show how fast you can dig into the low level details of an unknown technology using F#, and how little code you need to write. Therefore, I keep the text of the article short, focusing just on the required steps. If you need more information about F# or Amazon S3, there are plenty of books and articles available. Just Google for them. Instead of numbering steps that I've gone through, I'll provide timing information: I started the exercise at about 17:00 CET on July 15th. It took me about 45 minutes to display a picture extracted from my S3 bucket. And all code was written in F#.

17:00. Getting the Amazon Web Services SDK

Amazon Web Services SDK, also known as AWS SDK, is available at http://aws.amazon.com/sdkfornet. Once downloaded, its installer copies binaries and samples, and optionally registers AWSSDK.dll in the GAC.

17:10. You can't go wrong with Reflector

Before firing a Visual Studio F# session, I checked the contents of the installed binaries and loaded into Red Gate .NET Reflector the only .NET assembly that I found: AWSSDK.dll. I could probably have managed without it, but it would have taken longer. After I created an S3 client session, I mostly relied on Intellisense support for F# in Visual Studio, but spending a few minutes looking at namespaces and classes hinted me about entry points. For example, I assumed that I would need to create an instance of AmazonS3Client and probably use classes from Amazon.S3.Model.

ExploringAmazonwithFsharp/Reflector.JPG

17:15. Opening an F# Interactive session

I started Visual Studio and immediately was able to use the comparative advantage of the F# environment. I didn't have to create a project - in fact, I didn't know what kind of project I would need to create if I had to: Should it be a console application? Or a Windows Forms program? I didn't know yet what to expect, all I wanted to get from an IDE is a Notepad where I could write code lines and execute them one by one. So F# was a very good fit for my intention: I created a new F# script file and started sending individual lines to an FSI window. The first lines referenced AWSSDK and imported the Amazon.S3 namespace that I expected to be useful (thanks Reflector):

#r "AWSSDK"
open Amazon.S3

Later I imported another namespace (Amazon.S3.Model), but these two lines above were enough to connect to the Amazon S3 storage.

17:20. Connecting to Amazon S3 and accessing a bucket

The fun part began. My guess was that since the AmazonS3Client class has a constructor with two string parameters, this is the one I should use, and assign parameters to the AWS key ID and secret key. F# Intellisense confirmed it. So here's my first call to Amazon S3 in F# (I removed the actual access key values).

let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY")

After a few seconds, F# Interactive came with a response:

val s3Client : AmazonS3Client

Now I had a mighty client with many interesting methods to try. The next logical step would be to list the available buckets. I only had one, so I could retrieve only the first element of the list.

let response = s3Client.ListBuckets()
response.Buckets

Here's the FSI output:

val it : System.Collections.Generic.List<model.s3bucket> =
  seq
    [Amazon.S3.Model.S3Bucket {BucketName = "abilov.com";
                               CreationDate = "to, 15 jul 2010 10:17:18 GMT";}]

Great! This looked like a real thing. Encouraged with a quick success, I wanted to start retrieving the bucket contents.

17:25. Failed attempt to retrieve bucket objects

The bucket collection was wrapped in a sequence, so I only needed to get its head to obtain the only bucket I had at S3:

let bucket = Seq.head response.Buckets

I thought in order to retrieve bucket objects, I had to send the obtained bucket to some method, but I was wrong. Actually, I didn't need to obtain a bucket at all - since I knew the name of the bucket, I could construct a ListObjectRequest from the name and send it to a ListObjects. But I didn't know about it 5 minutes ago. Anyway, here're new calls, to retrieve bucket objects - data files with images from our home photo gallery:

let request = new ListObjectsRequest(BucketName = bucket.BucketName)
let objects = s3Client.ListObjects(request)

Bang!!! Exception!

System.Net.WebException: The underlying connection was closed: Could not establish 
        trust relationship for the SSL/TLS secure channel. ---> 
        System.Security.Authentication.AuthenticationException: 
        The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, 
      AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(
      ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, 
      Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, 
      Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, 
      Int32 readBytes, AsyncProtocolRequest asyncRequest)

(The rest of the call stack is skipped)

Now when I am writing this article, I am in fact glad that I had this problem. If everything went smooth, you might look at my timing information with a grain of disbelief: the guy probably knew what he was doing. No I didn't! So I had to check around. Since the error came after some basic steps, I hoped that I wasn't alone in experiencing it, and luckily, I was not. Several people complained at Amazon forum, and they were recommended a simple workaround in case they would accept using HTTP instead of HTTPS. And for the purpose of exploration, it was just fine.

17:30. Connection to Amazon S3 revisited

Here's an updated version of the connection code, taking now two lines instead of one.

let s3Config = new AmazonS3Config(CommunicationProtocol = Protocol.HTTP)
let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY", s3Config)

Then I replayed the other two lines requesting bucket objects. And now, I successfully received results:

{AmazonId2 = "//VkK6Mb7Xe26pNPG7nt40GkYcp4GDyNV45I1E1mfZVCL67aak9/0Txmrk8X+JAE";
CommonPrefix = seq [];
CommonPrefixes = seq [];
Delimiter = null;
Headers = seq
         ["x-amz-id-2"; "x-amz-request-id"; "Transfer-Encoding";
              "Content-Type"; ...];
IsTruncated = true;
MaxKeys = "1000";
Metadata = seq [];
Name = "abilov.com";
NextMarker = "private/photo/2005/2005-06-10-12 Elverum/IMG_6803.JPG";
Prefix = "";
RequestId = "3A27D1A4495A747A";
ResponseStream = null;
S3Objects = seq {…}

(The rest of results is skipped)

17:35. Retrieving object data

I inspected the response and found the property that I needed: S3Objects. It was a collection of bucket objects representing the photo gallery files I uploaded to S3. Taking a further look at the available AmazonS3Client methods, I found a method that I should use to retrieve an individual object: GetObject. It took an instance of GetObjectRequest. I first created a GetObjectRequest instance without assigning a Key property. But after receiving an exception with an error message about missing a key, I corrected my mistake. Here's a code that worked:

let objectRequest = new GetObjectRequest(BucketName=bucket.BucketName, 
          Key="private/photo/2005/2005-01 Kolbotn/IMG_5645.JPG")
let obj = s3Client.GetObject(objectRequest)

The "Key" property is just a file path within the bucket, so it's easy to figure it out.

I sent "obj" to an FSI window, and it printed its properties:

{AmazonId2 = "zUH8IaRglPSPQuC+8m5pzAR59paJTi/L5M1DnxNympp9HcU5RigrntS7NGvpxQQf";
ContentLength = 1138630L;
ContentType = "image/jpeg";
ETag = ""ea02192ee842d3b1987a8de4ac879595"";
Headers = ?;
Metadata = seq ["x-amz-meta-cb-modifiedtime"];
RequestId = "A93CF70A966E4A56";
ResponseStream = System.Net.ConnectStream;
ResponseXml = null;
VersionId = null;}

One of the properties was called ReponseStream. This sounded very promising - perhaps I could try to create an image out of this stream and display it in a Windows Form?

17:40. Displaying an image

What should you do if you want to display an image in a Windows Form in C#? You have to start from creating a new project using the Windows Forms Application template. Then you can edit the Form class and add an Image control to it in the Form designer. How do you do this in F#? You simply create a form with an image, right from the script. Here's the code:

#r "System.Drawing"
#r "System.Windows.Forms"
 
open System.Windows.Forms
open System.Drawing

let img = Image.FromStream(obj.ResponseStream)
let frm = new Form(ClientSize = img.Size)
frm.Paint.Add(fun e -> e.Graphics.DrawImage(img, new Point(0,0)))
frm.Show()

Note that the first four lines are just to reference the required DLLs and import namespaces. The rest of the code is equally short and completes the task. Look at the Image.FromStream call: I wasn't quite sure I could do this, but the property name "ResponseStream" was too tempting not to give it a try. And suddenly I saw this window:

ExploringAmazonwithFsharp/Figaro.JPG

This is our house cat Figaro (unfortunately, not with us anymore). And no, I didn't specifically select this photo for my test. This was a random choice, perhaps hinting about what kind of pictures some families tend to take.

17:45. Done!

Yes, we are! Perhaps just to show my environmentalist attitude, I'll make one last call:

s3Client.Dispose()

Conclusion

So it was three quarters of exploration. Completely new API, combined with my relatively poor experience with Amazon S3, but a picture from a family photo gallery retrieved from an S3 bucket and shown in a Windows Form proves that we managed the full stack of operations. All done in F#. This experiment does not expose other (and far more important) language qualities, but I deliberately wanted to focus on a different topic: language efficiency for the purpose of technology exploration and rapid prototyping. Being a .NET language, F# can access any .NET area or feature, and its REPL support ensures that the developer's time is spent in a most optimal way.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Vagif Abilov
Architect Miles AS
Norway Norway
Vagif Abilov is a Russian/Norwegian software developer and architect working for Miles. He has more than twenty years of programming experience that includes various programming languages, currently using mostly C# and F#.
 
Vagif writes articles and speaks at user group sessions and conferences. He is a contributor and maintainer of several open source projects, including Simple.Data OData adapter, Simple.OData.Client and MongOData.
Follow on   Twitter

Comments and Discussions

 
GeneralMy vote of 4 Pinmembertec-goblin25-Jul-10 0:24 
GeneralRe: My vote of 4 PinmemberVagif Abilov25-Jul-10 6:25 
GeneralRe: My vote of 4 Pinmembertec-goblin25-Jul-10 6:39 
GeneralRe: My vote of 4 PinmemberVagif Abilov25-Jul-10 7:09 
GeneralExcellent article PinmemberPatrick.Hartnett23-Jul-10 6:59 
GeneralRe: Excellent article PinmemberVagif Abilov24-Jul-10 4:02 
GeneralMy vote of 5 PinmemberEmile van Gerwen19-Jul-10 23:24 
GeneralRe: My vote of 5 PinmemberVagif Abilov24-Jul-10 4:02 
GeneralAmazon S3 freeware client Pinmembercloudberryman15-Jul-10 23:42 
GeneralRe: Amazon S3 freeware client PinmemberVagif Abilov16-Jul-10 2:34 
GeneralГут! PinmemberBashir Magomedov15-Jul-10 23:35 
GeneralMy vote of 5 Pinmemberdvhh15-Jul-10 16:41 
GeneralRe: My vote of 5 PinmemberVagif Abilov15-Jul-10 20:34 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.141220.1 | Last Updated 15 Jul 2010
Article Copyright 2010 by Vagif Abilov
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid