Today we're continuing our look at Microsoft's Cloud Computing initiative but posting a bit out of order. Microsoft has introduced some changes to Table Storage that I'm trying to address so Part 4 will be posted tomorrow. In the meantime, I'm posting Part 5 on Queue Storage.
In previous posts I covered an Introduction to Microsoft Azure, the way that Azure Storage handles Security and started yesterday with an Introduction to Azure Table Storage. Today I'm going to look at the next step, Azure Table storage. If you want to follow along, download a copy of my AzureCommand class. You also might want to create an Microsoft Azure Account. I should state that I am not looking at the locally hosted development storage, only at the cloud hosted one.
When one thinks Queues, one may conjure up many different images. Microsoft has MMQ and has implemented Service Broker, both as means of having stateless communication. Azure Queue Storage (AQS) is similar. The stated reason for using AQS is to be able to kick off Azure Worker Roles, but you can actually use AQS for any type of stateless communication. AQS consists of two parts: Queues and Messages. Queues, like Tables in ATS, are really nothing more than defined URIs that can be used to store and retrieve messages. Messages are nothing more than text packets (up to 8KB) that can be stored and retrieved from a specific URI. So, let's dig in.
Queues
Queues in AQS, are defined URIs. Queues can only exist at the root, so the URI will look like
http://{account}.queue.core.windows.net/{queue}
Queue names have some minor requirements. The queue name must be a valid DNS name, conforming to the following naming rules:
- A queue name must start with a letter or number, and may contain only letters, numbers and the dash (-) character.
- The first and last letters in the queue name must be alphanumeric. The dash (-) character may not be the first or last letter.
- All letters in a queue name must be lowercase.
- A queue name must be from 3 through 63 characters long
Failing to follow these rules will result in getting a status code 400 (Bad Request) or 403 (Forbidden), depending on how malformed the request is. For instance, attempting to create a queue called "-invalid-queue-name" will return a 400 (Bad Request) but "-invalid queue-name" returns 403 (Forbidden). This is an inconsistency that will cause you to tear your hair out trying to track down an authentication bug that isn't!
There are three things you can do to a Queue: Create a new Queue (POST), delete an existing Queue (DELETE) and get information about an existing Queue (GET). In addition, you can save and update Metadata associated with the Queue.
Creating a Queue
Creating a Queue is easy (as long as it is named correctly). Execute a POST against http://{account}.queue.core.windows.net/{queue} (CanonicalUrl is /{account}/{queue}). You can include Metadata headers (see below) but they are not required. Successfully creating a Queue returns a 201 (Created).
Getting Information About a Queue
Getting information about a Queue is easy. Execute a GET or a HEAD against http://{account}.queue.core.windows.net/{queue} (CanonicalUrl is /{account}/{queue}). The difference between HEAD and GET is that responses to HEAD calls never return a body and are more efficient, especially since all of the data about a Queue is in the headers. In this case, there are two types of headers we care about. One is x-ms-approximate-messages-count, which returns an approximation of the number of messages in the Queue (more about that below in the section on Messages). The other is any header that begins with x-ms-meta-, which is how AQS stores and returns Metadata.
Deleting a Queue
Deleting a Queue is easy. Execute a DELETE against http://{account}.queue.core.windows.net/{queue} (CanonicalUrl is /{account}/{queue}). You can include Metadata headers (see below) but they are not required. Successfully creating a Queue returns a 204 (No Content).
NOTE: Deleting a Queue deletes all messages in the Queue. You are given no warning that messages exist in the Queue at the time you delete it!
Best Practice: Always do a HEAD on the Queue and verify that the x-ms-approximate-messages-count is 0 before deleting the queue.
Queue Metadata
Queue Metadata is handled through HttpHeaders. Any header you create that begins with x-ms-meta- will be treated as Metadata and returned when you execute a GET or a HEAD against the Queue. All you can do is replace all of the Metadata on the Queue with a new set by executing a POST against http://{account}.queue.core.windows.net/{queue} (CanonicalUrl is /{account}/{queue}). My sample application and web interface both show a Delete Metadata but really I'm just executing the POST with a blank set of Metadata, which effectively removes all Metadata headers.
Messages
Messages are 8KB text stored in a FIFO manner (First In, First Out). There are no constraints on what you can put in the message other than size. If you want to use more than 8KB for your messaging, you should consider storing the extra data somewhere else, perhaps as a Blob in Azure Blob Storage (Part 6 of this series). There are five things you can do with a Message: Create a new message (POST), delete an existing Message (DELETE), get an existing message to process (GET), look at messages in the queue without disturbing them and delete all of the messages in the queue.
Creating a Message
Execute a POST against http://{account}.queue.core.windows.net/{queue}/messages (CanonicalUrl is /{account}/{queue}/messages) with the message in the body. If you are successful, you'll get a 201 (Created). If your message is larger than 8KB you'll get a 413 (Request Entity Too Large).
Getting a Message
Execute a GET against http://{account}.queue.core.windows.net/{queue}/messages (CanonicalUrl is /{account}/{queue}/messages). If you are successful you'll get a 200 (OK) and the message data will be in the body. If there is no message, the body will be empty. There are two optional parameters you can pass in that are not included in the CanonicalUrl. These are VisibilityTimeout and NumOfMessages (number of messages).
When you execute a Get Message you will either get an empty string (no messages) or a block back that looks like this:
xml version="1.0" encoding="utf-16"?> <QueueMessagesList><QueueMessage><MessageId>81f33dd9-9832-4c6d-aa26-5f0d4a972959MessageId><InsertionTime>Wed, 03 Jun 2009 11:49:08 GMTInsertionTime><ExpirationTime>Wed, 10 Jun 2009 11:48:28 GMTExpirationTime><PopReceipt>AQAAADAXmE1B5MkBPopReceipt><TimeNextVisible>Wed, 03 Jun 2009 11:49:08 GMTTimeNextVisible><MessageText>asdfasfdMessageText>QueueMessage> QueueMessagesList>
If you specified a value on the NumOfMessages, you can have up to that number of multiple QueueMessage elements to process.
Getting a message doesn't actually remove it from the queue. What it does is set a parameter called the Visibility Timeout. This determines how long it will be until the message appears on the queue again. The TimeNextVisible basically determines how long you have to work on this queue item and delete it before it shows up in the queue again. The default is 30 seconds, but you can change that by using a parameter on URI you are calling (this does NOT change the canonical URI) named visibilitytimeout. So calling http://{account}.queue.core.windows.net/{queue}/messages?visibiltytimeout=90 will result in getting a message that won't show up in the queue for 90 seconds. The maximum value you can set for a message to not show up on the queue is 2 hours.
Looking at Messages (Peeking)
There are two differences between Looking at Messages and Getting Messages. The first is that Looking requires you to pass in a parameter (not included in the CanonicalUrl) of peekonly=true so you are executing a GET against http://{account}.queue.core.windows.net/{queue}/messages?peekonly=true (CanonicalUrl is /{account}/{queue}/messages). The second is that the XML will contain neitehr MessageId nor PopReceipt for any of the messages. But it does let you see what's in the queue. This can be combined with the numofmessages parameter discussed in getting messages below.
xml version="1.0" encoding="utf-8"?> <QueueMessagesList><QueueMessage><MessageId>1bb1c258-cca3-42cd-8332-8d9630286aafMessageId><InsertionTime>Wed, 03 Jun 2009 12:25:20 GMTInsertionTime><ExpirationTime>Wed, 10 Jun 2009 12:25:20 GMTExpirationTime><MessageText>This is a message to processMessageText>QueueMessage> QueueMessagesList>
Deleting a Message
Deleting a message requires two pieces of information from the QueueMessage block retrieved when you get a message, the MessageID and the PopReceipt. The MessageID will always be associated with that message. If you Get a message, wait until it is visible again and get it again, you'll see that the MessageID stays the same. The PopReceipt, however, changes. The MessageID is actually used in creating the URI to post against and the PopReceipt is included as a parameter.
Execute a DELETE against http://{account}.queue.core.windows.net/{queue}/messages/{messageId}?popreceipt={popReceipt} (CanonicalUrl is /{account}/{queue}/messages/{messageId}). If you are successful you'll get a 204 (No Content). If the message has already been deleted or someone else has also retrieved the message, you'll get a 404 (Not Found).
Note: I'd rather I got a 409 Conflict rather than a 404 Not Found. A Not Found implies the message has been deleted. A Conflict implies that the Message still exists but my PopReceipt is not good.
I haven't fully thought through the implications of that last statement but let's say you get a message from the queue and it takes you 45 seconds to process it rather than the 30 you expect it to. That means that someone else is going to get that message and will start processing it, so it is best to make sure anything you do that is based on queues is Idempotent, that way, two queues processing the same message won't screw anything up.
Wrapping up
That pretty much covers everything related to Azure Queue Storage. Next week, after I've finished with Table and Blob Storage and begin to talk about Worker and Web Roles, I'll talk more about Queues as I look into how to handle some of the management of data that RDBMS does for us today to make better use of Azure Table Storage.
All Posts in Series: