A REST-ful Look at Azure (Part 7) Blob Storage continued

Jun9

Written by:CarpDeus
6/9/2009 10:18 AM

So far, this series has covered an Introduction to Microsoft Azure, the way that Azure Storage handles Security, an Introduction to Azure Table Storage, more on Azure Table Storage and Azure Queue Storage. Yesterday I started into Azure Blob Storage, but only covered the simple parts of loading a Blob. Loading Blobs larger than 64MB requires special processing. Today I'm going to look at both that and how my web version handles processing the file without creating a disk file. If you want to follow along, download a copy of my AzureCommand class. You also might want to create an Microsoft Azure Account. I should state that I am not looking at the locally hosted development storage, only at the cloud hosted one.

Creating and Retrieving Blobs with Blocks

If you want to upload a large file to Azure Blob Storage (ABS), you're going to need to split it into pieces and transmit the individual pieces. Then, when you get them all uploaded, you tell the server to assemble them. This is all done using Blocks, which may be up to 4MB in size. It sounds complicated, but it isn't really. The first thing you do is split the file into 4MB blocks. If you have a Byte Array, this is easy enough to do. Then, you upload each block to the server using PUT and the URI of http://{account}.blob.corewindows.net/{Container}/{Blob}?comp=block&blockid={blockid} (CanonicalURL is {account}/{Container}/{Blob}?comp=block). BlockID must be a valid Base64 string that describes the block. Oh, and all of the BlockIDs must be the same length. For a true idea of what that entails, this is the entire PutBlob function from the class:

   1:public azureResults PutBlob(Int64 ContentLength, string ContentType, byte[] Content, string containerName, string blobName, Hashtable htMetaData)

   2:  {

   3:   azureResults retVal = new azureResults();

   4:try

   5:   {

   6:    StringBuilder sb = new StringBuilder();

   7:string sendBody = string.Empty;

   8:string rtnBody = string.Empty;

9:

  10:string requestUrl = string.Format(CultureInfo.CurrentCulture, "{0}{1}/{2}", auth.EndPoint, containerName, blobName);

  11:    requestDate = DateTime.UtcNow;

  12:if (ContentLength < maxNonBlockBytes)

  13:    {

  14:     HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(requestUrl);

  15:     req.Method = "PUT";

  16:     req.ContentType = ContentType;

  17:     req.ContentLength = ContentLength;

  18:if (htMetaData.Count > 0)

  19:foreach (DictionaryEntry item in htMetaData)

  20:      {

  21:string metaDataName = item.Key.ToString().ToLower().Replace(" ", "-").Replace("\r", "");

  22:if (!metaDataName.StartsWith("x-ms-meta-"))

  23:        metaDataName = "x-ms-meta-" + metaDataName;

  24:try

  25:       {

  26:if (item.Value.ToString().Trim() != string.Empty)

  27:         req.Headers[metaDataName] = item.Value.ToString();

  28:       }

  29:catch (Exception ex) { }

  30:      }

  31:     req.Headers["x-ms-date"] = string.Format(CultureInfo.CurrentCulture, "{0:R}", requestDate);

  32:     retVal.CanonicalResource = ac.CanonicalizeUrl(requestUrl);

  33:     authHeader = ac.CreateSharedKeyAuth(req.Method, retVal.CanonicalResource, contentMD5, requestDate, req, auth);

  34:     req.Headers["authorization"] = authHeader;

35:

  36:     Stream requestStream = req.GetRequestStream();

  37:     requestStream.Write(Content, 0, (int)ContentLength);

  38:     requestStream.Flush();

  39:     HttpWebResponse response = (HttpWebResponse)req.GetResponse();

  40:     response.Close();

41:

  42:     retVal.Url = requestUrl;

  43:     retVal.Body = "";

  44:     retVal.StatusCode = response.StatusCode;

  45:     retVal.Headers = ac.Headers2Hash(response.Headers);

  46:     retVal.Succeeded = (retVal.StatusCode == HttpStatusCode.Created);

  47:    }

  48:else

  49:    {

  50:string blockURI = string.Empty;

  51:     Hashtable htHeaders = htMetaData;

  52:     htHeaders.Add("Content-Type", contentType);

  53:     azureDirect ad = new azureDirect(auth.Account, auth.EndPoint, auth.SharedKey, auth.KeyType);

  54:int blocksCount = (int)Math.Ceiling((double)ContentLength / maxNonBlockBytes);

  55:string[] blockIds = new string[blocksCount];

  56:int startPosition = 0;

  57:for (int i = 0; i < blocksCount; i++)

  58:     {

  59:      blockIds[i] = Convert.ToBase64String(BitConverter.GetBytes(i));

  60:      blockURI = string.Format("{0}?comp=block&blockid={1}", requestUrl, blockIds[i]);

  61:byte[] blockContent = new byte[maxNonBlockBytes];

  62:      Array.Copy(Content, startPosition, blockContent, 0, (startPosition + maxNonBlockBytes > Content.Length ? Content.Length - startPosition : maxNonBlockBytes));

  63:      retVal = ad.ProcessRequest(cmdType.put, blockURI, blockContent, htHeaders);

  64:      startPosition += maxNonBlockBytes;

  65:     }

  66:     blockURI = string.Format("{0}?comp=blocklist", requestUrl);

  67:     StringBuilder sbBlockList = new StringBuilder();

  68:     sbBlockList.Append("\n");

  69:foreach (string id in blockIds)

  70:     {

  71:      sbBlockList.AppendFormat("{0}\n", id);

  72:     }

  73:     sbBlockList.Append("");

  74:     retVal = ad.ProcessRequest(cmdType.put, blockURI, new System.Text.ASCIIEncoding().GetBytes(sbBlockList.ToString()), htHeaders);

  75:    }

  76:   }

  77:catch (HttpException hex)

  78:   {

  79:    retVal.StatusCode = (HttpStatusCode)hex.GetHttpCode();

  80:    retVal.Succeeded = false;

  81:    retVal.Body = hex.GetHtmlErrorMessage();

  82:   }

  83:catch (Exception ex)

  84:   {

  85:    retVal.StatusCode = HttpStatusCode.SeeOther;

  86:    retVal.Body = ex.ToString();

  87:    retVal.Succeeded = false;

  88:   }

  89:return retVal;

  90:  }

The first thing to notice is that I'm using a Byte Array to hold the body. One reason for that is that a Byte Array is a handy way to pass files back and forth. Not only that but, if I have an array large enough that I need to pull pieces out to transmit blocks, I can easily just take a portion of the Array and use it.

Lines 12 through 47 cover the basics of putting a Blob that's small enough to not require the use of Blocks. The single interesting item there is lines 36 through 40, which covers how I send the actual item to Azure. First, I get a copy of the httpRequest object's Stream. Next, I write the Byte Array to it. Finally, I flush the httpRequest's stream and execute the request's GetResponse method. That's all there is to it.

If the object is larger than 64 MB it needs to be loaded in 4MB blocks. For that reason, I simply load anything larger than 4MB in 4MB blocks. This is the code in lines 49-74. The two key sections are 57-65 where I post each individual block and 69-74 where I post the list of blocks and commit the object. The basic process for posting the blocks is to get the BlockID and add it to both the string array I'll use to commit the blocks and to the URI I need to PUT the data to. Next I get a Byte Array that contains the set of bytes I want to post. Finally, I post that byte array as a Block to http://{account}.blob.core.windows.net/{Container}/{Blob}?comp=block&blockid={blockid} (CanonicalUrl is /{account}/{Container}/{Blob}?comp=block.)

Then, I walk through the string array of BlockIDs and build an XML representation of the blocks and PUT that in the body to http://{account}.blob.core.windows.net/{Container}/{Blob}?comp=blocklist (CanonicalUrl is /{account}/{Container}/{Blob}?comp=blocklist.) If all goes well, I'll get a status of 201 (Created).

Processing in a WinApp

Putting blobs to ABS is super simple from a WinApp. I have a simple function that takes a file name and turns it into a byte array:

public byte[] FileToByteArray(string fileName)

byte[] retVal = null;

try

FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read);

retVal = new BinaryReader(fs).ReadBytes((int)fs.Length);

catch (Exception ex)

Console.WriteLine(ex.Message);

return retVal;

The tricky part with a WinApp is getting the ContentType, but that's relatively simple given that the information is stored in the registry:

private string getMimeType(string fileName, string defaultMimeType)

string mimeType = defaultMimeType ;

string ext = System.IO.Path.GetExtension(fileName).ToLower();

Microsoft.Win32.RegistryKey regKey = Microsoft.Win32.Registry.ClassesRoot.OpenSubKey(ext);

if (regKey != null && regKey.GetValue("Content Type") != null)

mimeType = regKey.GetValue("Content Type").ToString();

return mimeType;

Using those two functions, I have all the information I need to store a Blob in ABS. And it's simpler on the web.

Processing in a WebForm

Putting blobs to ABS is super simple from a WebForm. First, we need the controls to load the file:

<asp:FileUpload ID="FileUpload1" runat="server" />

<asp:Button ID="Button3" runat="server" OnClick="Button3_Click" Text="Upload File" />

Next, we add the code behind for it:

protected void Button3_Click(object sender, EventArgs e)

if (FileUpload1.HasFile)

try

string containerName = (txtContainer.Text == string.Empty ? cbContainers.Text : txtContainer.Text);

string filename = FileUpload1.FileName;

byte[] blobArray =  FileUpload1.FileBytes;

Finsel.AzureCommands.AzureBlobStorage abs = new Finsel.AzureCommands.AzureBlobStorage(txtAccount.Text, string.Format("http://{0}.blob.core.windows.net", txtAccount.Text), txtSharedKey.Text, "SharedKey");

azureResults ar = abs.PutBlob(blobArray.Length, FileUpload1.PostedFile.ContentType,

blobArray, containerName, FileUpload1.FileName, new Hashtable());

ProcessResults(ar);

catch (Exception ex) { }

else  { }

The best part about the web version is that we can access the Byte Array directly from the FileUpload object and it can tell us the Content Type. That's as easy as it gets, we just pass the data in.

Copying a Blob

There's one last piece to cover with Blobs, and that is a new piece of functionality that allows you to copy them. To do this, you execute a PUT against http://{account}.blob.corewindows.net/{Container}/{Blob} (CanonicalURL is {account}/{Container}/{Blob}) This is the new blob you are creating. Then, in the httpRequest headers you include the CanonicalUrl of the blob you are copying with as the x-ms-copy-source. That's all there is to it. I've implemented a CopyBlob function in the class that takes care of this for you:

public azureResults CopyBlob(string OriginalBlobUrl, string NewBlobUrl, Hashtable htHeaders)

if (htHeaders.ContainsKey("x-ms-copy-source"))

htHeaders["x-ms-copy-source"] = new azureCommon().CanonicalizeUrl(OriginalBlobUrl);

else

htHeaders.Add("x-ms-copy-source", new azureCommon().CanonicalizeUrl(OriginalBlobUrl));

azureResults retVal = new azureResults();

azureDirect ad = new azureDirect(auth);

retVal = ad.ProcessRequest(cmdType.put, NewBlobUrl, "", htHeaders);

return retVal;

In addition, there are numerous options you can set in the headers so that it will only copy if it matches certain criteria (see API for more details). Unless you specify the Content Type or MetaData (using the x-ms-meta- headers), it will copy over the content type and metadata on the original blob.

And that wraps up Blobs. Tomorrow we'll look at Azure's Hosted offerings and then we'll dig into making Azure Table Storage more useful using all of these tools.