Click here to Skip to main content
15,845,103 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
Hi,

I am working on a distributed application with Azure Storage Queue for message queuing. queue will be used by multiple clients across the clock and thus it is expected that it would be heavily loaded most on the time in usage. business case is typical as in it pulls message from queue, process the message then deletes the message from queue. this module also sends back a notification to user indicating process is complete. functions/modules work fine as in they meet the logical requirement. pretty typical queue scenario.

Now, coming to the problem statement. since it is envisaged that the queue would be heavily loaded most of the time, I am pushing towards to speed up processing of the overall message lifetime. the faster I can clear messages, the better overall experience it would be for everyone, system and users.

To improve on performance I did multiple cycles for performance profiling and then improving on the identified "HOT" path/function.
It all came down to a point where only the Azure Queue pull and delete are the only two most time consuming calls outside. I can further improve on pull, which i did by batch pulling 35 message at a time (which is the max message count i can pull from Azure queue at once at the time of writing this question.), this returned me a favor as in by reducing processing time to a big margin. all good till this as well.
i am processing these messages in parallel so as to improve on overall performance.

pseudo code:
C#
//AzureQueue Class is encapsulating calls to Azure Storage Queue.
//assume nothing fancy inside, vanila calls to queue for pull/push/delete
var batchMessages = AzureQueue.Pull(35);            Parallel.ForEach(batchMessages, bMessage =>
            {
              //DoSomething does some background processing;
              try{DoSomething(bMessage);}
              catch()
              {
               //Log exception
              }
              AzureQueue.Remove(bMessage);
            });


With this change now, profiling results show that up-to 90% of time is only taken by the Azure Message delete calls. As it is good to delete message as soon as processing is done, i remove it just after "DoSomething" is finished.

what i need now is suggestions on how to further improve performance of this function when 90% of the time is being eaten up by the Azure Queue Delete call itself? is there a better faster way to perform delete/bulk delete etc?

with the implementation mentioned here, i get speed of close to 25 messages/sec. Right now Azure queue delete calls are choking application performance. so is there any hope to push it further.

Let me know if you need any additional information or any clarification in question.

Inputs/suggestions are welcome.

Many thanks.
Posted
Updated 30-Oct-14 3:01am
v3
Comments
girishkalamati 27-Nov-14 7:07am    
r u able to resolve this issue ?

let me try to help u out on this any way its a very good question ....
OmniSource 1-Apr-15 7:06am    
Hi Girish,

Issue still exist however it's in cold bucket cause of other priorities with project.
deployed this package on Azure and throughput was close to 18~20 message per sec.
each message is about 64kb, size can't be reduced further.

as it stands, it's a time bomb ticking, will have to go back to it once time demands.

Thanks for comments.

1 solution

hi OmniSource

If you come to the Performance increase in azure queues service , at first thought Service Bus Queue would be a prominent answer for this ?
but i was wrong and also right in some case

please go through this link

my answer would be if you can comprmoise in decreasing the Message Size itself then it can do alot to WAKE up the PERFORMANCE !

as you can increase the
Quote:
throughput up to 2,000 messages per second
(based on benchmark with 1 KB messages)
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900