Click here to Skip to main content
Click here to Skip to main content

Beginning with Amazon S3

By , 24 Apr 2011
Rate this:
Please Sign up or sign in to vote.

Introduction

Amazon S3 (Amazon Simple Storage Service) is a service that allows to store files online. This article is about how a beginner can develop applications with Amazon S3 using C#.

Amazon S3 can help us store data as files using a folder structure, similar to an online hard disk. We can create files, folders, upload a file, delete a file/folder, etc. We will talk about how this can be done using C# from a developer's point of view.

The first thing, you need is the Amazon Web Service SDK. Secondly, you need an Amazon Web Service account with an access key and private key to connect to Amazon S3.

Services

Create an AmazonS3 object

Now let us start with Amazon S3. We need an Amazon S3 account, access key, and private key, to create an AmazonS3 object. This object controls all the actions to interact with the Amazon S3 server.

const string AWS_ACCESS_KEY = "put_your_AWS_access_key_here";
const string AWS_SECRET_KEY = "put_your_AWS_secret_key _here";
AmazonS3 client = new AmazonS3(AWS_ACCESS_KEY, AWS_SECRET_KEY)

Normally, these keys are stored in the “web.config” and we access them by code:

// In application config
<appSettings>
    <add key="AWSAccessKey" value="put_your_AWS_access_key_here"/>
    <add key="AWSSecretKey" value="put_your_AWS_secret_key _here"/>
<appSettings>

The code:

// Function to get it
public static AmazonS3 GetS3Client()
{
    NameValueCollection appConfig = ConfigurationManager.AppSettings;

    AmazonS3 s3Client = AWSClientFactory.CreateAmazonS3Client(
            appConfig["AWSAccessKey"],
            appConfig["AWSSecretKey"]
            );
    return s3Client;
}

Create a bucket

If you want to store data in S3, you need to create a bucket. It is similar to a root folder in Windows. In Amazon S3, the maximum number of buckets is 100 and the names of buckets are unique globally.

string BUCKET_NAME = "stormspiritbucket1";
ListBucketsResponse response = client.ListBuckets();
bool found = false;
foreach (S3Bucket bucket in response.Buckets)
{
    if (bucket.BucketName == BUCKET_NAME)
    {                   
        found = true;
        break;
    }
}
if (found == false)
{
    client.PutBucket(new PutBucketRequest().WithBucketName(BUCKET_NAME));
}

After that, in the Amazon S3 console, you will see this bucket:

Add a new file to Amazon S3

The first thing to do to store a file is adding the file to the Amazon S3 server. There are many ways to do it, and I will show two of the most popular ways.

  1. Create a file in Amazon S3:
  2. To create a file in Amazon S3, we need the following information:

    • FileKey: It can be understood as the file name, but it is more specific. It must be unique and should include the folder path, like a physical address in Windows.
    • ContentBody: The content of the new file.
    String S3_KEY = "Demo Create File.txt";
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);
    request.WithContentBody("This is body of S3 object.");
    client.PutObject(request);

    The new file will be created as a text file.

    You want to create a folder in S3? No problem, we use the FileKey with the character '/' at the end to show that you want to create a folder.

    String S3_KEY = "Demo Create folder/";
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);
    request.WithContentBody("");
    client.PutObject(request);

    And this is the result:

    If you want to create a file in the folder, you need change the file key to include the folder name:

    String S3_KEY = "Demo Create folder/" + "Demo Create File.txt";
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);
    request.WithContentBody("This is body of S3 object.");
    client.PutObject(request);

    And this is the result:

    However, a problem when you create a file with the key "TestFolder/File.txt" is the folder 'TestFolder' does not exist. When you do this, in the Amazon S3 console, you will see a folder 'TestFolder' and this folder will contain the file 'File.txt'. But the S3 database will not contain the folder 'TestFolder', and if you delete the file 'File.txt', the folder 'TestFolder' will be deleted too. You cannot get any item from S3 with the key 'TestFolder/'.

  3. Upload files to Amazon S3:
  4. This method is to upload a file from a local computer to Amazon S3. Of course, we need a file key to upload a file. We can upload a local file using a file path, like this:

    //S3_KEY is name of file we want to upload
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);
    request.WithFilePath(pathToFile)
    client.PutObject(request);

    And we can upload a file from a stream too:

    //S3_KEY is name of file we want to upload
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);
    request.WithInputStream(ms);
    client.PutObject(request);

Get a file from Amazon S3

When you want to get a file from S3, use the method GetObject:

using (s3Client)
{
    MemoryStream file = new MemoryStream();
    try
    {
        GetObjectResponse r = s3Client.GetObject(new GetObjectRequest()
        {
            BucketName = bucketName,
            Key = fileKey
        });            
        try
        {
            long transferred = 0L;
            BufferedStream stream2 = new BufferedStream(r.ResponseStream);
            byte[] buffer = new byte[0x2000];
            int count = 0;
            while ((count = stream2.Read(buffer, 0, buffer.Length)) > 0)
            {
                file.Write(buffer, 0, count);
            }
        }
        finally
        {
        }
        return file;
    }
    catch (AmazonS3Exception)
    { 
        //Show exception
    }
}

If you want to list all items in S3 or in a specific folder, use the method listObject:

using (s3Client)
{
    try
    {        
        ListObjectsRequest Lor = new ListObjectsRequest()
        {
            BucketName = bucketName,
            // with Prefix is a folder Key, it will list only child of that folder
            Prefix = folderPath, 
            //with Delimiter is '/', it will not get folder.
            Delimiter = "/"
        };        
        ListObjectsResponse response1 = s3Client.ListObjects(Lor);
        foreach (S3Object s3Object in response1.S3Objects)
        { 
            //Do anything you want
        }
    }
    catch (AmazonS3Exception ex)
    {
        //Show Exception
    }
}

Delete a file from Amazon S3

When you want to delete an item in S3, you need the exact key for the item.

DeleteObjectRequest request = new DeleteObjectRequest()
{
    BucketName = bucketName,
    Key = FileKey
};
S3Response response = s3Client.DeleteObject(request);

Unfortunately, when deleting a folder, Amazon S3 does not support deleting its children. Therefore, this is similar to the case of creating a file in a non-existing folder. It may be a real problem, so you need to handle it with your own code. The best solution I know in this case is to list all the ‘children’ of your folder and delete them one by one.

Copy and paste

Amazon S3 supports copy and paste of files from a source to a destination:

CopyObjectRequest request = new CopyObjectRequest()
{
    SourceBucket = bucketName,
    SourceKey = fileKey,
    DestinationBucket = bucketName,
    DestinationKey = destinationPath
};
CopyObjectResponse response = s3Client.CopyObject(request);

Share file

If you have a file in Amazon S3 that you want to share with others, there are two ways to doing it:

  1. Change file permission:
  2. You know that the default permission when you create a file in S3 is private, which means only you can view and edit the file. Therefore, you need to change the permission of the file to public to share the file with others. Amazon S3 supports these permissions: AuthenticatedRead, BucketOwnerFullControl, BucketOwnerRead, NoACL, Private (default), PublicRead (share with others), PublicReadWrite.

    S3Response response1 = s3Client.SetACL(new SetACLRequest()
    {
        CannedACL = permission,
        BucketName = bucketName,
        Key = keyName
    });
  3. Make a public URL:
  4. Amazon S3 supports creating a public URL (PreSignedURL). With this link, a public user access a file for reading for a specified period. After the period, the link will be disposes. This is the most popular way to share files in S3.

    string preSignedURL = s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
    {                        
        BucketName = bucketName,
        Key = keyName,
        Expires = System.DateTime.Now.AddMinutes(period)
    });

Conclusion

Amazon Simple Storage Service has a lot of useful features and is very useful for anyone who wants to develop an application that need to be store data online. It has the structure and UI similar to Windows apps, and with the Amazon Web Service SDK, beginners get an easy start.

History

  • 21 Apr, 2011: Initial post.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

StormSpirit Team
Student FPT University
Vietnam Vietnam
StormSpirit Team:
HiepNT - 00403
HuyHN - 00320
DucNM - 00303
ThangTS - 00098
Supervisor: Professor Nguyen Hong Ky.

Comments and Discussions

 
QuestionHow to check if a folder exists in a specific bucket ? Pinmembermanoj0409199015-Jul-13 17:15 
GeneralMy vote of 4 PinmemberLazaro Lima4-Feb-13 12:16 
QuestionAWS S3, VB.NET PinmemberKevinBrady5724-Nov-12 6:21 
GeneralMy vote of 5 PinmvpKanasz Robert27-Sep-12 8:28 
QuestionGood Article Pinmemberelshiekh511-Sep-12 3:09 
GeneralAmazon s3 PinmemberJames8493916-Jul-11 9:50 
GeneralMy vote of 5 PinmemberKim Togo25-Apr-11 21:37 
Good article about S3 storage Smile | :) - Bookmarked for later use.
GeneralAmazon S3 freeware client Pinmembercloudberryman24-Apr-11 22:14 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140415.2 | Last Updated 24 Apr 2011
Article Copyright 2011 by StormSpirit Team
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid