Beginning with Amazon S3

• Download demo project - 14.2 KB

Introduction

Amazon S3 (Amazon Simple Storage Service) is a service that allows to store files online. This article is about how a beginner can develop applications with Amazon S3 using C#.

Amazon S3 can help us store data as files using a folder structure, similar to an online hard disk. We can create files, folders, upload a file, delete a file/folder, etc. We will talk about how this can be done using C# from a developer's point of view.

The first thing, you need is the Amazon Web Service SDK. Secondly, you need an Amazon Web Service account with an access key and private key to connect to Amazon S3.

Services

Create an AmazonS3 object

Now let us start with Amazon S3. We need an Amazon S3 account, access key, and private key, to create an AmazonS3 object. This object controls all the actions to interact with the Amazon S3 server.

const string AWS_ACCESS_KEY = "put_your_AWS_access_key_here";
const string AWS_SECRET_KEY = "put_your_AWS_secret_key _here";
AmazonS3 client = new AmazonS3(AWS_ACCESS_KEY, AWS_SECRET_KEY)

Normally, these keys are stored in the “web.config” and we access them by code:

// In application config
<appSettings>
    <add key="AWSAccessKey" value="put_your_AWS_access_key_here"/>
    <add key="AWSSecretKey" value="put_your_AWS_secret_key _here"/>
<appSettings>

The code:

// Function to get it
public static AmazonS3 GetS3Client()
{
    NameValueCollection appConfig = ConfigurationManager.AppSettings;

    AmazonS3 s3Client = AWSClientFactory.CreateAmazonS3Client(
            appConfig["AWSAccessKey"],
            appConfig["AWSSecretKey"]
            );
    return s3Client;
}

Create a bucket

If you want to store data in S3, you need to create a bucket. It is similar to a root folder in Windows. In Amazon S3, the maximum number of buckets is 100 and the names of buckets are unique globally.

string BUCKET_NAME = "stormspiritbucket1";
ListBucketsResponse response = client.ListBuckets();
bool found = false;
foreach (S3Bucket bucket in response.Buckets)
{
    if (bucket.BucketName == BUCKET_NAME)
    {                   
        found = true;
        break;
    }
}
if (found == false)
{
    client.PutBucket(new PutBucketRequest().WithBucketName(BUCKET_NAME));
}

After that, in the Amazon S3 console, you will see this bucket:

Add a new file to Amazon S3

The first thing to do to store a file is adding the file to the Amazon S3 server. There are many ways to do it, and I will show two of the most popular ways.

1. Create a file in Amazon S3:

To create a file in Amazon S3, we need the following information:

• FileKey: It can be understood as the file name, but it is more specific. It must be unique and should include the folder path, like a physical address in Windows.
• ContentBody: The content of the new file.
C#

String S3_KEY = "Demo Create File.txt";
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName(BUCKET_NAME);
request.WithKey(S3_KEY);
request.WithContentBody("This is body of S3 object.");
client.PutObject(request);

The new file will be created as a text file.



You want to create a folder in S3? No problem, we use the FileKey with the character '/' at the end to show that you want to create a folder.

C#

String S3_KEY = "Demo Create folder/";
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName(BUCKET_NAME);
request.WithKey(S3_KEY);
request.WithContentBody("");
client.PutObject(request);

And this is the result:



If you want to create a file in the folder, you need change the file key to include the folder name:

C#

String S3_KEY = "Demo Create folder/" + "Demo Create File.txt";
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName(BUCKET_NAME);
request.WithKey(S3_KEY);
request.WithContentBody("This is body of S3 object.");
client.PutObject(request);

And this is the result:



However, a problem when you create a file with the key "TestFolder/File.txt" is the folder 'TestFolder' does not exist. When you do this, in the Amazon S3 console, you will see a folder 'TestFolder' and this folder will contain the file 'File.txt'. But the S3 database will not contain the folder 'TestFolder', and if you delete the file 'File.txt', the folder 'TestFolder' will be deleted too. You cannot get any item from S3 with the key 'TestFolder/'.

2. Upload files to Amazon S3:

This method is to upload a file from a local computer to Amazon S3. Of course, we need a file key to upload a file. We can upload a local file using a file path, like this:

C#

//S3_KEY is name of file we want to upload
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName(BUCKET_NAME);
request.WithKey(S3_KEY);
request.WithFilePath(pathToFile)
client.PutObject(request);

And we can upload a file from a stream too:

C#

//S3_KEY is name of file we want to upload
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName(BUCKET_NAME);
request.WithKey(S3_KEY);
request.WithInputStream(ms);
client.PutObject(request);

Get a file from Amazon S3

When you want to get a file from S3, use the method GetObject:

using (s3Client)
{
    MemoryStream file = new MemoryStream();
    try
    {
        GetObjectResponse r = s3Client.GetObject(new GetObjectRequest()
        {
            BucketName = bucketName,
            Key = fileKey
        });            
        try
        {
            long transferred = 0L;
            BufferedStream stream2 = new BufferedStream(r.ResponseStream);
            byte[] buffer = new byte[0x2000];
            int count = 0;
            while ((count = stream2.Read(buffer, 0, buffer.Length)) > 0)
            {
                file.Write(buffer, 0, count);
            }
        }
        finally
        {
        }
        return file;
    }
    catch (AmazonS3Exception)
    { 
        //Show exception
    }
}

If you want to list all items in S3 or in a specific folder, use the method listObject:

using (s3Client)
{
    try
    {        
        ListObjectsRequest Lor = new ListObjectsRequest()
        {
            BucketName = bucketName,
            // with Prefix is a folder Key, it will list only child of that folder
            Prefix = folderPath, 
            //with Delimiter is '/', it will not get folder.
            Delimiter = "/"
        };        
        ListObjectsResponse response1 = s3Client.ListObjects(Lor);
        foreach (S3Object s3Object in response1.S3Objects)
        { 
            //Do anything you want
        }
    }
    catch (AmazonS3Exception ex)
    {
        //Show Exception
    }
}

Delete a file from Amazon S3

When you want to delete an item in S3, you need the exact key for the item.

DeleteObjectRequest request = new DeleteObjectRequest()
{
    BucketName = bucketName,
    Key = FileKey
};
S3Response response = s3Client.DeleteObject(request);

Unfortunately, when deleting a folder, Amazon S3 does not support deleting its children. Therefore, this is similar to the case of creating a file in a non-existing folder. It may be a real problem, so you need to handle it with your own code. The best solution I know in this case is to list all the ‘children’ of your folder and delete them one by one.

Copy and paste

Amazon S3 supports copy and paste of files from a source to a destination:

CopyObjectRequest request = new CopyObjectRequest()
{
    SourceBucket = bucketName,
    SourceKey = fileKey,
    DestinationBucket = bucketName,
    DestinationKey = destinationPath
};
CopyObjectResponse response = s3Client.CopyObject(request);

Share file

If you have a file in Amazon S3 that you want to share with others, there are two ways to doing it:

1. Change file permission:

You know that the default permission when you create a file in S3 is private, which means only you can view and edit the file. Therefore, you need to change the permission of the file to public to share the file with others. Amazon S3 supports these permissions: AuthenticatedRead, BucketOwnerFullControl, BucketOwnerRead, NoACL, Private (default), PublicRead (share with others), PublicReadWrite.

C#

S3Response response1 = s3Client.SetACL(new SetACLRequest()
{
    CannedACL = permission,
    BucketName = bucketName,
    Key = keyName
});

2. Make a public URL:

Amazon S3 supports creating a public URL (PreSignedURL). With this link, a public user access a file for reading for a specified period. After the period, the link will be disposes. This is the most popular way to share files in S3.

C#

string preSignedURL = s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
{                        
    BucketName = bucketName,
    Key = keyName,
    Expires = System.DateTime.Now.AddMinutes(period)
});

Conclusion

Amazon Simple Storage Service has a lot of useful features and is very useful for anyone who wants to develop an application that need to be store data online. It has the structure and UI similar to Windows apps, and with the Amazon Web Service SDK, beginners get an easy start.

History

• 21 Apr, 2011: Initial post.