Loading ...
Sorry, an error occurred while loading the content.

Re: [altdotnet] Clustering Azure Queue?

Expand Messages
  • Shawn Hinsey
    Have you looked at the Azure AppFabric Queues? They have different performance characteristics.
    Message 1 of 20 , Sep 26, 2011
    View Source
    • 0 Attachment
      Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

      On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

      Hi guys,
      I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
      I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
      Thanks before


    • Hendry Luk
      I had looked at it briefly at it, and it seems to be focused around features we don t actually need (i.e. pub/sub), and the pricing could be quite expensive to
      Message 2 of 20 , Sep 26, 2011
      View Source
      • 0 Attachment
        I had looked at it briefly at it, and it seems to be focused around features we don't actually need (i.e. pub/sub), and the pricing could be quite expensive to be used within a scalable system (involving hundreds of worker instances). But I'll investigate its performance characteristics, and see if it's any better than azure-queue. But so far I can't see anything about clustering/partitioning support either.
        Can you give more pointer of how appfabric queue scales?

        Thanks

        On Tue, Sep 27, 2011 at 1:24 PM, Shawn Hinsey <smhinsey@...> wrote:
         

        Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

        On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

        Hi guys,
        I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
        I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
        Thanks before



      • Shawn Hinsey
        To be honest, I don t know much more about the appfabric queues than what is publicly available. I m going to be facing a similar question myself shortly, but
        Message 3 of 20 , Sep 26, 2011
        View Source
        • 0 Attachment
          To be honest, I don't know much more about the appfabric queues than what is publicly available. I'm going to be facing a similar question myself shortly, but I haven't gotten there just yet. If you like, I can reach out to my contacts at MS and see what they have to say. Just shoot me an email off-list with any additional info you might want to share and I will pass it along.

          On Tuesday, September 27, 2011 at 1:51 AM, Hendry Luk wrote:

          I had looked at it briefly at it, and it seems to be focused around features we don't actually need (i.e. pub/sub), and the pricing could be quite expensive to be used within a scalable system (involving hundreds of worker instances). But I'll investigate its performance characteristics, and see if it's any better than azure-queue. But so far I can't see anything about clustering/partitioning support either.
          Can you give more pointer of how appfabric queue scales?

          Thanks

          On Tue, Sep 27, 2011 at 1:24 PM, Shawn Hinsey <smhinsey@...> wrote:
           

          Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

          On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

          Hi guys,
          I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
          I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
          Thanks before




        • Simone Busoli
          Hi Hendry, never worked with Azure, but I have doubts it has been designed with this throughput in mind. Have you considered alternatives like RabbitMQ or
          Message 4 of 20 , Sep 27, 2011
          View Source
          • 0 Attachment

            Hi Hendry, never worked with Azure, but I have doubts it has been designed with this throughput in mind. Have you considered alternatives like RabbitMQ or ZeroMQ? Those would probably handle your load way better.

            On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
            > Hi guys,
            > I'm developing an application with an expected SLA of 10,000 transactions
            > per second. From load-tests, I discovered azure-queue to be the bottleneck
            > of the system, where it can only seem to handle a terrifyingly low 60tps.
            > I've been trying to look into clustering/load-balancing the queue, with no
            > much joy so far. Has anyone had any experience/advice on the scalability
            > story around azure-queue? How can I scale it up to achieve anywhere near the
            > kind of throughput I need?
            > Thanks before
          • Hendry Luk
            Thanks Shawn, I ve sent to your email. The current approach I ve tried is to write a rudimentary round-robin logic in our hub to spread the load across
            Message 5 of 20 , Sep 27, 2011
            View Source
            • 0 Attachment
              Thanks Shawn, I've sent to your email.

              The current approach I've tried is to write a rudimentary round-robin logic in our hub to spread the load across multiple azure storage accounts. I've load-tested on a simple prototype across 4 queue storage accounts, and found that the troughput has improved almost proportionally. To achieve the desired throughput, it seems that I will therefore need to create few hundreds of storage accounts. But I'm wondering if there's any better approach to this.

              I've been in touch with MS as well, haven't  come up with a definite solution yet, but would love to hear the experiences from other people here.
              Cheers

              On Tue, Sep 27, 2011 at 4:06 PM, Shawn Hinsey <smhinsey@...> wrote:
               

              To be honest, I don't know much more about the appfabric queues than what is publicly available. I'm going to be facing a similar question myself shortly, but I haven't gotten there just yet. If you like, I can reach out to my contacts at MS and see what they have to say. Just shoot me an email off-list with any additional info you might want to share and I will pass it along.

              On Tuesday, September 27, 2011 at 1:51 AM, Hendry Luk wrote:

              I had looked at it briefly at it, and it seems to be focused around features we don't actually need (i.e. pub/sub), and the pricing could be quite expensive to be used within a scalable system (involving hundreds of worker instances). But I'll investigate its performance characteristics, and see if it's any better than azure-queue. But so far I can't see anything about clustering/partitioning support either.
              Can you give more pointer of how appfabric queue scales?

              Thanks

              On Tue, Sep 27, 2011 at 1:24 PM, Shawn Hinsey <smhinsey@...> wrote:
               

              Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

              On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

              Hi guys,
              I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
              I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
              Thanks before





            • Hendry Luk
              Thanks Simon. I wonder if any of those MQ solutions works (properly) on azure.
              Message 6 of 20 , Sep 27, 2011
              View Source
              • 0 Attachment
                Thanks Simon. I wonder if any of those MQ solutions works (properly) on azure.

                On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...> wrote:
                 

                Hi Hendry, never worked with Azure, but I have doubts it has been designed with this throughput in mind. Have you considered alternatives like RabbitMQ or ZeroMQ? Those would probably handle your load way better.

                On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                > Hi guys,
                > I'm developing an application with an expected SLA of 10,000 transactions
                > per second. From load-tests, I discovered azure-queue to be the bottleneck
                > of the system, where it can only seem to handle a terrifyingly low 60tps.
                > I've been trying to look into clustering/load-balancing the queue, with no
                > much joy so far. Has anyone had any experience/advice on the scalability
                > story around azure-queue? How can I scale it up to achieve anywhere near the
                > kind of throughput I need?
                > Thanks before


              • Simone Busoli
                I honestly don t know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating
                Message 7 of 20 , Sep 27, 2011
                View Source
                • 0 Attachment

                  I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                  I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                  On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                  > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                  > azure.
                  >
                  > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                  >
                  >> **
                  >>
                  >>
                  >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                  >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                  >> or ZeroMQ? Those would probably handle your load way better.
                  >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                  >> > Hi guys,
                  >> > I'm developing an application with an expected SLA of 10,000 transactions
                  >> > per second. From load-tests, I discovered azure-queue to be the
                  >> bottleneck
                  >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                  >> > I've been trying to look into clustering/load-balancing the queue, with
                  >> no
                  >> > much joy so far. Has anyone had any experience/advice on the scalability
                  >> > story around azure-queue? How can I scale it up to achieve anywhere near
                  >> the
                  >> > kind of throughput I need?
                  >> > Thanks before
                  >>
                  >>
                  >>
                • Hendry Luk
                  I ve also just demystified the pricing scheme of azure-queue, and apparently you ll pay per each message-polling. During off-peak time, having hundreds of
                  Message 8 of 20 , Sep 27, 2011
                  View Source
                  • 0 Attachment
                    I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                    Thanks for all the suggestions.

                    On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                     

                    I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                    I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                    On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                    > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                    > azure.
                    >
                    > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                    >
                    >> **
                    >>
                    >>
                    >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                    >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                    >> or ZeroMQ? Those would probably handle your load way better.
                    >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                    >> > Hi guys,
                    >> > I'm developing an application with an expected SLA of 10,000 transactions
                    >> > per second. From load-tests, I discovered azure-queue to be the
                    >> bottleneck
                    >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                    >> > I've been trying to look into clustering/load-balancing the queue, with
                    >> no
                    >> > much joy so far. Has anyone had any experience/advice on the scalability
                    >> > story around azure-queue? How can I scale it up to achieve anywhere near
                    >> the
                    >> > kind of throughput I need?
                    >> > Thanks before
                    >>
                    >>
                    >>


                  • kelly.leahy@milliman.com
                    Hendry, Have you tried just using different queues within the same storage account? I don t see any reason why you d need to put the queues in separate
                    Message 9 of 20 , Sep 27, 2011
                    View Source
                    • 0 Attachment

                      Hendry,

                       

                      Have you tried just using different queues within the same storage account?  I don’t see any reason why you’d need to put the queues in separate accounts to see the benefits of load balancing, but maybe I’m wrong.  It is definitely not true that you need to separate blobs by storage account to receive the benefits of “hot blob” load balancing.

                       

                      Kelly

                       

                      From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk <hendrymail@...>
                      Sent: Tuesday, September 27, 2011 12:59 AM
                      To: altdotnet@yahoogroups.com
                      Subject: Re: [altdotnet] Clustering Azure Queue?

                       

                       

                       

                      Thanks Shawn, I've sent to your email.

                      The current approach I've tried is to write a rudimentary round-robin logic in our hub to spread the load across multiple azure storage accounts. I've load-tested on a simple prototype across 4 queue storage accounts, and found that the troughput has improved almost proportionally. To achieve the desired throughput, it seems that I will therefore need to create few hundreds of storage accounts. But I'm wondering if there's any better approach to this.

                      I've been in touch with MS as well, haven't  come up with a definite solution yet, but would love to hear the experiences from other people here.
                      Cheers

                      On Tue, Sep 27, 2011 at 4:06 PM, Shawn Hinsey <smhinsey@...> wrote:

                       

                      To be honest, I don't know much more about the appfabric queues than what is publicly available. I'm going to be facing a similar question myself shortly, but I haven't gotten there just yet. If you like, I can reach out to my contacts at MS and see what they have to say. Just shoot me an email off-list with any additional info you might want to share and I will pass it along.

                      On Tuesday, September 27, 2011 at 1:51 AM, Hendry Luk wrote:

                      I had looked at it briefly at it, and it seems to be focused around features we don't actually need (i.e. pub/sub), and the pricing could be quite expensive to be used within a scalable system (involving hundreds of worker instances). But I'll investigate its performance characteristics, and see if it's any better than azure-queue. But so far I can't see anything about clustering/partitioning support either.
                      Can you give more pointer of how appfabric queue scales?

                      Thanks

                      On Tue, Sep 27, 2011 at 1:24 PM, Shawn Hinsey <smhinsey@...> wrote:

                       

                      Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

                      On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

                      Hi guys,
                      I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
                      I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
                      Thanks before

                       

                       

                       

                       


                      **************************************************************************************
                      This communication is intended solely for the addressee and is
                      confidential. If you are not the intended recipient, any disclosure,
                      copying, distribution or any action taken or omitted to be taken in
                      reliance on it, is prohibited and may be unlawful. Unless indicated
                      to the contrary: it does not constitute professional advice or opinions
                      upon which reliance may be made by the addressee or any other party,
                      and it should be considered to be a work in progress. Unless otherwise
                      noted in this email or its attachments, this communication does not form
                      a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                      **************************************************************************************
                    • Hendry Luk
                      I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs. 1. If I change my producer
                      Message 10 of 20 , Sep 28, 2011
                      View Source
                      • 0 Attachment
                        I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                        1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                        2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                        3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                        Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                        NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                        On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:
                        I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                        Thanks for all the suggestions.


                        On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                         

                        I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                        I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                        On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                        > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                        > azure.
                        >
                        > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                        >
                        >> **
                        >>
                        >>
                        >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                        >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                        >> or ZeroMQ? Those would probably handle your load way better.
                        >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                        >> > Hi guys,
                        >> > I'm developing an application with an expected SLA of 10,000 transactions
                        >> > per second. From load-tests, I discovered azure-queue to be the
                        >> bottleneck
                        >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                        >> > I've been trying to look into clustering/load-balancing the queue, with
                        >> no
                        >> > much joy so far. Has anyone had any experience/advice on the scalability
                        >> > story around azure-queue? How can I scale it up to achieve anywhere near
                        >> the
                        >> > kind of throughput I need?
                        >> > Thanks before
                        >>
                        >>
                        >>



                      • kelly.leahy@milliman.com
                        Hendry, have you looked at AzureScope? http://azurescope.cloudapp.net/BenchmarkTestCases/tc/4a02b847-1078-4413-922f-b7b7cd52d681/ Kelly ... To:
                        Message 11 of 20 , Sep 28, 2011
                        View Source
                        • 0 Attachment
                          Hendry, have you looked at AzureScope?
                           
                           
                          Kelly
                          -----altdotnet@yahoogroups.com wrote: -----

                          To: altdotnet@yahoogroups.com
                          From: Hendry Luk <hendrymail@...>
                          Sent by: altdotnet@yahoogroups.com
                          Date: 09/28/2011 06:30PM
                          Subject: Re: [altdotnet] Clustering Azure Queue?

                           

                          I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                          1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                          2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                          3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                          Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                          NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                          On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:
                          I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                          Thanks for all the suggestions.


                          On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                           

                          I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                          I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                          On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                          > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                          > azure.
                          >
                          > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                          >
                          >> **
                          >>
                          >>
                          >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                          >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                          >> or ZeroMQ? Those would probably handle your load way better.
                          >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                          >> > Hi guys,
                          >> > I'm developing an application with an expected SLA of 10,000 transactions
                          >> > per second. From load-tests, I discovered azure-queue to be the
                          >> bottleneck
                          >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                          >> > I've been trying to look into clustering/load-balancing the queue, with
                          >> no
                          >> > much joy so far. Has anyone had any experience/advice on the scalability
                          >> > story around azure-queue? How can I scale it up to achieve anywhere near
                          >> the
                          >> > kind of throughput I need?
                          >> > Thanks before
                          >>
                          >>
                          >>





                          **************************************************************************************
                          This communication is intended solely for the addressee and is
                          confidential. If you are not the intended recipient, any disclosure,
                          copying, distribution or any action taken or omitted to be taken in
                          reliance on it, is prohibited and may be unlawful. Unless indicated
                          to the contrary: it does not constitute professional advice or opinions
                          upon which reliance may be made by the addressee or any other party,
                          and it should be considered to be a work in progress. Unless otherwise
                          noted in this email or its attachments, this communication does not form
                          a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                          **************************************************************************************
                        • kelly.leahy@milliman.com
                          Oh... in case it wasn t clear from AzureScope, I recommend setting the ServicePoint.DefaultMaxConnectionLimit to 20 and see what happens in your case #2 below.
                          Message 12 of 20 , Sep 28, 2011
                          View Source
                          • 0 Attachment
                            Oh... in case it wasn't clear from AzureScope, I recommend setting the ServicePoint.DefaultMaxConnectionLimit to 20 and see what happens in your case #2 below.
                             
                            Kelly
                            -----altdotnet@yahoogroups.com wrote: -----

                            To: altdotnet@yahoogroups.com
                            From: Hendry Luk <hendrymail@...>
                            Sent by: altdotnet@yahoogroups.com
                            Date: 09/28/2011 06:30PM
                            Subject: Re: [altdotnet] Clustering Azure Queue?

                             

                            I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                            1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                            2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                            3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                            Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                            NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                            On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:
                            I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                            Thanks for all the suggestions.


                            On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                             

                            I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                            I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                            On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                            > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                            > azure.
                            >
                            > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                            >
                            >> **
                            >>
                            >>
                            >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                            >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                            >> or ZeroMQ? Those would probably handle your load way better.
                            >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                            >> > Hi guys,
                            >> > I'm developing an application with an expected SLA of 10,000 transactions
                            >> > per second. From load-tests, I discovered azure-queue to be the
                            >> bottleneck
                            >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                            >> > I've been trying to look into clustering/load-balancing the queue, with
                            >> no
                            >> > much joy so far. Has anyone had any experience/advice on the scalability
                            >> > story around azure-queue? How can I scale it up to achieve anywhere near
                            >> the
                            >> > kind of throughput I need?
                            >> > Thanks before
                            >>
                            >>
                            >>





                            **************************************************************************************
                            This communication is intended solely for the addressee and is
                            confidential. If you are not the intended recipient, any disclosure,
                            copying, distribution or any action taken or omitted to be taken in
                            reliance on it, is prohibited and may be unlawful. Unless indicated
                            to the contrary: it does not constitute professional advice or opinions
                            upon which reliance may be made by the addressee or any other party,
                            and it should be considered to be a work in progress. Unless otherwise
                            noted in this email or its attachments, this communication does not form
                            a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                            **************************************************************************************
                          • Hendry Luk
                            No I had not. Thank you for that.
                            Message 13 of 20 , Oct 4, 2011
                            View Source
                            • 0 Attachment
                              No I had not. Thank you for that.

                              On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:
                               

                              Hendry, have you looked at AzureScope?
                               
                               
                              Kelly
                              -----altdotnet@yahoogroups.com wrote: -----

                              To: altdotnet@yahoogroups.com
                              From: Hendry Luk <hendrymail@...>
                              Sent by: altdotnet@yahoogroups.com
                              Date: 09/28/2011 06:30PM
                              Subject: Re: [altdotnet] Clustering Azure Queue?


                               

                              I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                              1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                              2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                              3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                              Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                              NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                              On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:
                              I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                              Thanks for all the suggestions.


                              On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                               

                              I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                              I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                              On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                              > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                              > azure.
                              >
                              > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                              >
                              >> **
                              >>
                              >>
                              >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                              >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                              >> or ZeroMQ? Those would probably handle your load way better.
                              >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                              >> > Hi guys,
                              >> > I'm developing an application with an expected SLA of 10,000 transactions
                              >> > per second. From load-tests, I discovered azure-queue to be the
                              >> bottleneck
                              >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                              >> > I've been trying to look into clustering/load-balancing the queue, with
                              >> no
                              >> > much joy so far. Has anyone had any experience/advice on the scalability
                              >> > story around azure-queue? How can I scale it up to achieve anywhere near
                              >> the
                              >> > kind of throughput I need?
                              >> > Thanks before
                              >>
                              >>
                              >>





                              **************************************************************************************
                              This communication is intended solely for the addressee and is
                              confidential. If you are not the intended recipient, any disclosure,
                              copying, distribution or any action taken or omitted to be taken in
                              reliance on it, is prohibited and may be unlawful. Unless indicated
                              to the contrary: it does not constitute professional advice or opinions
                              upon which reliance may be made by the addressee or any other party,
                              and it should be considered to be a work in progress. Unless otherwise
                              noted in this email or its attachments, this communication does not form
                              a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                              **************************************************************************************


                            • Hendry Luk
                              I believe you can only have one queue storage on each storage account. (Though you can have multiple queue-name, which is akin to table in a rdbms). ... I
                              Message 14 of 20 , Oct 4, 2011
                              View Source
                              • 0 Attachment
                                I believe you can only have one queue storage on each storage account. (Though you can have multiple queue-name, which is akin to table in a rdbms).

                                On Tue, Sep 27, 2011 at 7:03 PM, <kelly.leahy@...> wrote:
                                 

                                Hendry,

                                 

                                Have you tried just using different queues within the same storage account?  I don’t see any reason why you’d need to put the queues in separate accounts to see the benefits of load balancing, but maybe I’m wrong.  It is definitely not true that you need to separate blobs by storage account to receive the benefits of “hot blob” load balancing.

                                 

                                Kelly

                                 

                                From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk <hendrymail@...>
                                Sent: Tuesday, September 27, 2011 12:59 AM
                                To: altdotnet@yahoogroups.com


                                Subject: Re: [altdotnet] Clustering Azure Queue?

                                 

                                 

                                 

                                Thanks Shawn, I've sent to your email.



                                The current approach I've tried is to write a rudimentary round-robin logic in our hub to spread the load across multiple azure storage accounts. I've load-tested on a simple prototype across 4 queue storage accounts, and found that the troughput has improved almost proportionally. To achieve the desired throughput, it seems that I will therefore need to create few hundreds of storage accounts. But I'm wondering if there's any better approach to this.

                                I've been in touch with MS as well, haven't  come up with a definite solution yet, but would love to hear the experiences from other people here.
                                Cheers

                                On Tue, Sep 27, 2011 at 4:06 PM, Shawn Hinsey <smhinsey@...> wrote:

                                 

                                To be honest, I don't know much more about the appfabric queues than what is publicly available. I'm going to be facing a similar question myself shortly, but I haven't gotten there just yet. If you like, I can reach out to my contacts at MS and see what they have to say. Just shoot me an email off-list with any additional info you might want to share and I will pass it along.

                                On Tuesday, September 27, 2011 at 1:51 AM, Hendry Luk wrote:

                                I had looked at it briefly at it, and it seems to be focused around features we don't actually need (i.e. pub/sub), and the pricing could be quite expensive to be used within a scalable system (involving hundreds of worker instances). But I'll investigate its performance characteristics, and see if it's any better than azure-queue. But so far I can't see anything about clustering/partitioning support either.
                                Can you give more pointer of how appfabric queue scales?

                                Thanks

                                On Tue, Sep 27, 2011 at 1:24 PM, Shawn Hinsey <smhinsey@...> wrote:

                                 

                                Have you looked at the Azure AppFabric Queues? They have different performance characteristics.

                                On Monday, September 26, 2011 at 10:03 PM, Hendry Luk wrote:

                                Hi guys,
                                I'm developing an application with an expected SLA of 10,000 transactions per second. From load-tests, I discovered azure-queue to be the bottleneck of the system, where it can only seem to handle a terrifyingly low 60tps.
                                I've been trying to look into clustering/load-balancing the queue, with no much joy so far. Has anyone had any experience/advice on the scalability story around azure-queue? How can I scale it up to achieve anywhere near the kind of throughput I need?
                                Thanks before

                                 

                                 

                                 

                                 


                                **************************************************************************************
                                This communication is intended solely for the addressee and is
                                confidential. If you are not the intended recipient, any disclosure,
                                copying, distribution or any action taken or omitted to be taken in
                                reliance on it, is prohibited and may be unlawful. Unless indicated
                                to the contrary: it does not constitute professional advice or opinions
                                upon which reliance may be made by the addressee or any other party,
                                and it should be considered to be a work in progress. Unless otherwise
                                noted in this email or its attachments, this communication does not form
                                a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                **************************************************************************************


                              • Hendry Luk
                                Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard
                                Message 15 of 20 , Oct 5, 2011
                                View Source
                                • 0 Attachment
                                  Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard evidence that AzureQueue is capable to enqueue at 750tps when I ran it against my account.
                                  Through reflector, I managed to track it down to Nagle algorithm (which the analyzer code disables). When I disable my own NagleAlgorithm, coupled with ramping up my threads (instead of using the default thread-pool configuration), I finally managed to reach the same level of throughput (750tps).
                                  Which is a massive improvement.

                                  It's still spectacularly low for a queue, but at least it surpasses AzureQueue's quoted throughput (500tps).
                                  To reach 10k tps, I'll definitely still need to round-robin across multiple storage accounts. I recently came across a project that does exactly that (http://partitioncloudqueue.codeplex.com/), although I haven't evaluated how it's any different from my own naiive 3 lines round-robin implementation that I'm currently using.

                                  Thanks for all the help

                                  On Tue, Oct 4, 2011 at 6:08 PM, Hendry Luk <hendrymail@...> wrote:
                                  No I had not. Thank you for that.


                                  On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:
                                   

                                  Hendry, have you looked at AzureScope?
                                   
                                   
                                  Kelly
                                  -----altdotnet@yahoogroups.com wrote: -----

                                  To: altdotnet@yahoogroups.com
                                  From: Hendry Luk <hendrymail@...>
                                  Sent by: altdotnet@yahoogroups.com
                                  Date: 09/28/2011 06:30PM
                                  Subject: Re: [altdotnet] Clustering Azure Queue?


                                   

                                  I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                                  1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                                  2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                                  3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                                  Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                                  NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                                  On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:
                                  I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                                  Thanks for all the suggestions.


                                  On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:
                                   

                                  I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                                  I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                                  On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                                  > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                                  > azure.
                                  >
                                  > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                                  >
                                  >> **
                                  >>
                                  >>
                                  >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                                  >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                                  >> or ZeroMQ? Those would probably handle your load way better.
                                  >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                                  >> > Hi guys,
                                  >> > I'm developing an application with an expected SLA of 10,000 transactions
                                  >> > per second. From load-tests, I discovered azure-queue to be the
                                  >> bottleneck
                                  >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                                  >> > I've been trying to look into clustering/load-balancing the queue, with
                                  >> no
                                  >> > much joy so far. Has anyone had any experience/advice on the scalability
                                  >> > story around azure-queue? How can I scale it up to achieve anywhere near
                                  >> the
                                  >> > kind of throughput I need?
                                  >> > Thanks before
                                  >>
                                  >>
                                  >>





                                  **************************************************************************************
                                  This communication is intended solely for the addressee and is
                                  confidential. If you are not the intended recipient, any disclosure,
                                  copying, distribution or any action taken or omitted to be taken in
                                  reliance on it, is prohibited and may be unlawful. Unless indicated
                                  to the contrary: it does not constitute professional advice or opinions
                                  upon which reliance may be made by the addressee or any other party,
                                  and it should be considered to be a work in progress. Unless otherwise
                                  noted in this email or its attachments, this communication does not form
                                  a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                  **************************************************************************************



                                • Michael Brown
                                  I think the general guidance is to use Azure App fabric for high throughput. Have you done any tests against that for performance? Also, you get billed on
                                  Message 16 of 20 , Oct 6, 2011
                                  View Source
                                  • 0 Attachment

                                    I think the general guidance is to use Azure App fabric for high throughput. Have you done any tests against that for performance? Also, you get billed on number of connections rather than per transaction which is also a better model for very high numbers.

                                     

                                    From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk
                                    Sent: Wednesday, October 05, 2011 3:18 AM
                                    To: altdotnet@yahoogroups.com
                                    Subject: Re: [altdotnet] Clustering Azure Queue?

                                     

                                     

                                    Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard evidence that AzureQueue is capable to enqueue at 750tps when I ran it against my account.
                                    Through reflector, I managed to track it down to Nagle algorithm (which the analyzer code disables). When I disable my own NagleAlgorithm, coupled with ramping up my threads (instead of using the default thread-pool configuration), I finally managed to reach the same level of throughput (750tps).
                                    Which is a massive improvement.

                                    It's still spectacularly low for a queue, but at least it surpasses AzureQueue's quoted throughput (500tps).
                                    To reach 10k tps, I'll definitely still need to round-robin across multiple storage accounts. I recently came across a project that does exactly that (http://partitioncloudqueue.codeplex.com/), although I haven't evaluated how it's any different from my own naiive 3 lines round-robin implementation that I'm currently using.

                                    Thanks for all the help

                                    On Tue, Oct 4, 2011 at 6:08 PM, Hendry Luk <hendrymail@...> wrote:

                                    No I had not. Thank you for that.

                                     

                                    On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:

                                     

                                    Hendry, have you looked at AzureScope?

                                     

                                     

                                    Kelly

                                    -----altdotnet@yahoogroups.com wrote: -----

                                    To: altdotnet@yahoogroups.com
                                    From: Hendry Luk <hendrymail@...>
                                    Sent by: altdotnet@yahoogroups.com
                                    Date: 09/28/2011 06:30PM
                                    Subject: Re: [altdotnet] Clustering Azure Queue?



                                     

                                    I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                                    1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                                    2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                                    3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                                    Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                                    NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                                    On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:

                                    I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                                    Thanks for all the suggestions.

                                     

                                    On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:

                                     

                                    I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                                    I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                                    On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                                    > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                                    > azure.
                                    >
                                    > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                                    >
                                    >> **
                                    >>
                                    >>
                                    >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                                    >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                                    >> or ZeroMQ? Those would probably handle your load way better.
                                    >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                                    >> > Hi guys,
                                    >> > I'm developing an application with an expected SLA of 10,000 transactions
                                    >> > per second. From load-tests, I discovered azure-queue to be the
                                    >> bottleneck
                                    >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                                    >> > I've been trying to look into clustering/load-balancing the queue, with
                                    >> no
                                    >> > much joy so far. Has anyone had any experience/advice on the scalability
                                    >> > story around azure-queue? How can I scale it up to achieve anywhere near
                                    >> the
                                    >> > kind of throughput I need?
                                    >> > Thanks before
                                    >>
                                    >>
                                    >>

                                     

                                     



                                    **************************************************************************************
                                    This communication is intended solely for the addressee and is
                                    confidential. If you are not the intended recipient, any disclosure,
                                    copying, distribution or any action taken or omitted to be taken in
                                    reliance on it, is prohibited and may be unlawful. Unless indicated
                                    to the contrary: it does not constitute professional advice or opinions
                                    upon which reliance may be made by the addressee or any other party,
                                    and it should be considered to be a work in progress. Unless otherwise
                                    noted in this email or its attachments, this communication does not form
                                    a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                    **************************************************************************************

                                     

                                     

                                  • Hendry Luk
                                    I forgot to get back. I tried switching to appfabric service-bus, and that thing is damn fast! I built a one-tenth scale model of my actual production
                                    Message 17 of 20 , Oct 22, 2011
                                    View Source
                                    • 0 Attachment
                                      I forgot to get back. I tried switching to appfabric service-bus, and that thing is damn fast! I built a one-tenth scale model of my actual production requirement, hitting more than 2000 writes and 2000 reads per seconds, and service-bus hasn't even break a sweat. This thing so far looks pretty invincible. I'm quite happy with it, and it's a lot a lot a lot more cost effective as well, just a fraction of what I would have paid to poll hundreds of queue storage accounts.
                                      Thanks to all who've suggested it.

                                      On Fri, Oct 7, 2011 at 6:40 AM, Michael Brown <mbrown@...> wrote:
                                       

                                      I think the general guidance is to use Azure App fabric for high throughput. Have you done any tests against that for performance? Also, you get billed on number of connections rather than per transaction which is also a better model for very high numbers.

                                       

                                      From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk
                                      Sent: Wednesday, October 05, 2011 3:18 AM
                                      To: altdotnet@yahoogroups.com


                                      Subject: Re: [altdotnet] Clustering Azure Queue?

                                       

                                       

                                      Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard evidence that AzureQueue is capable to enqueue at 750tps when I ran it against my account.
                                      Through reflector, I managed to track it down to Nagle algorithm (which the analyzer code disables). When I disable my own NagleAlgorithm, coupled with ramping up my threads (instead of using the default thread-pool configuration), I finally managed to reach the same level of throughput (750tps).
                                      Which is a massive improvement.

                                      It's still spectacularly low for a queue, but at least it surpasses AzureQueue's quoted throughput (500tps).
                                      To reach 10k tps, I'll definitely still need to round-robin across multiple storage accounts. I recently came across a project that does exactly that (http://partitioncloudqueue.codeplex.com/), although I haven't evaluated how it's any different from my own naiive 3 lines round-robin implementation that I'm currently using.

                                      Thanks for all the help

                                      On Tue, Oct 4, 2011 at 6:08 PM, Hendry Luk <hendrymail@...> wrote:

                                      No I had not. Thank you for that.

                                       

                                      On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:

                                       

                                      Hendry, have you looked at AzureScope?

                                       

                                       

                                      Kelly

                                      -----altdotnet@yahoogroups.com wrote: -----

                                      To: altdotnet@yahoogroups.com
                                      From: Hendry Luk <hendrymail@...>
                                      Sent by: altdotnet@yahoogroups.com
                                      Date: 09/28/2011 06:30PM
                                      Subject: Re: [altdotnet] Clustering Azure Queue?



                                       

                                      I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                                      1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                                      2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                                      3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                                      Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                                      NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                                      On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:

                                      I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                                      Thanks for all the suggestions.

                                       

                                      On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:

                                       

                                      I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                                      I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                                      On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                                      > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                                      > azure.
                                      >
                                      > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                                      >
                                      >> **
                                      >>
                                      >>
                                      >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                                      >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                                      >> or ZeroMQ? Those would probably handle your load way better.
                                      >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                                      >> > Hi guys,
                                      >> > I'm developing an application with an expected SLA of 10,000 transactions
                                      >> > per second. From load-tests, I discovered azure-queue to be the
                                      >> bottleneck
                                      >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                                      >> > I've been trying to look into clustering/load-balancing the queue, with
                                      >> no
                                      >> > much joy so far. Has anyone had any experience/advice on the scalability
                                      >> > story around azure-queue? How can I scale it up to achieve anywhere near
                                      >> the
                                      >> > kind of throughput I need?
                                      >> > Thanks before
                                      >>
                                      >>
                                      >>

                                       

                                       



                                      **************************************************************************************
                                      This communication is intended solely for the addressee and is
                                      confidential. If you are not the intended recipient, any disclosure,
                                      copying, distribution or any action taken or omitted to be taken in
                                      reliance on it, is prohibited and may be unlawful. Unless indicated
                                      to the contrary: it does not constitute professional advice or opinions
                                      upon which reliance may be made by the addressee or any other party,
                                      and it should be considered to be a work in progress. Unless otherwise
                                      noted in this email or its attachments, this communication does not form
                                      a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                      **************************************************************************************

                                       

                                       


                                    • Hendry Luk
                                      I forgot to get back. I tried switching to appfabric service-bus, and that thing is damn fast! I built a one-tenth scale model of my actual production
                                      Message 18 of 20 , Oct 22, 2011
                                      View Source
                                      • 0 Attachment
                                        I forgot to get back.
                                        I tried switching to appfabric service-bus, and that thing is damn fast! I built a one-tenth scale model of my actual production requirement, hitting more than 2000 writes and 2000 reads per second, and service-bus hasn't even broken a sweat yet (when all my worker VMs have already reached 100% cpu capacity at this point). This thing so far looks pretty invincible. I'm quite happy with it, and it's a lot a lot a lot more cost effective as well, just a tiny fraction of what I would have paid to poll hundreds of queue storage accounts.
                                        Thanks to all who've suggested it.

                                        On Fri, Oct 7, 2011 at 6:40 AM, Michael Brown <mbrown@...> wrote:
                                         

                                        I think the general guidance is to use Azure App fabric for high throughput. Have you done any tests against that for performance? Also, you get billed on number of connections rather than per transaction which is also a better model for very high numbers.

                                         

                                        From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk
                                        Sent: Wednesday, October 05, 2011 3:18 AM
                                        To: altdotnet@yahoogroups.com


                                        Subject: Re: [altdotnet] Clustering Azure Queue?

                                         

                                         

                                        Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard evidence that AzureQueue is capable to enqueue at 750tps when I ran it against my account.
                                        Through reflector, I managed to track it down to Nagle algorithm (which the analyzer code disables). When I disable my own NagleAlgorithm, coupled with ramping up my threads (instead of using the default thread-pool configuration), I finally managed to reach the same level of throughput (750tps).
                                        Which is a massive improvement.

                                        It's still spectacularly low for a queue, but at least it surpasses AzureQueue's quoted throughput (500tps).
                                        To reach 10k tps, I'll definitely still need to round-robin across multiple storage accounts. I recently came across a project that does exactly that (http://partitioncloudqueue.codeplex.com/), although I haven't evaluated how it's any different from my own naiive 3 lines round-robin implementation that I'm currently using.

                                        Thanks for all the help

                                        On Tue, Oct 4, 2011 at 6:08 PM, Hendry Luk <hendrymail@...> wrote:

                                        No I had not. Thank you for that.

                                         

                                        On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:

                                         

                                        Hendry, have you looked at AzureScope?

                                         

                                         

                                        Kelly

                                        -----altdotnet@yahoogroups.com wrote: -----

                                        To: altdotnet@yahoogroups.com
                                        From: Hendry Luk <hendrymail@...>
                                        Sent by: altdotnet@yahoogroups.com
                                        Date: 09/28/2011 06:30PM
                                        Subject: Re: [altdotnet] Clustering Azure Queue?



                                         

                                        I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                                        1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                                        2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                                        3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                                        Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                                        NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                                        On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:

                                        I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                                        Thanks for all the suggestions.

                                         

                                        On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:

                                         

                                        I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                                        I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                                        On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                                        > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                                        > azure.
                                        >
                                        > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                                        >
                                        >> **
                                        >>
                                        >>
                                        >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                                        >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                                        >> or ZeroMQ? Those would probably handle your load way better.
                                        >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                                        >> > Hi guys,
                                        >> > I'm developing an application with an expected SLA of 10,000 transactions
                                        >> > per second. From load-tests, I discovered azure-queue to be the
                                        >> bottleneck
                                        >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                                        >> > I've been trying to look into clustering/load-balancing the queue, with
                                        >> no
                                        >> > much joy so far. Has anyone had any experience/advice on the scalability
                                        >> > story around azure-queue? How can I scale it up to achieve anywhere near
                                        >> the
                                        >> > kind of throughput I need?
                                        >> > Thanks before
                                        >>
                                        >>
                                        >>

                                         

                                         



                                        **************************************************************************************
                                        This communication is intended solely for the addressee and is
                                        confidential. If you are not the intended recipient, any disclosure,
                                        copying, distribution or any action taken or omitted to be taken in
                                        reliance on it, is prohibited and may be unlawful. Unless indicated
                                        to the contrary: it does not constitute professional advice or opinions
                                        upon which reliance may be made by the addressee or any other party,
                                        and it should be considered to be a work in progress. Unless otherwise
                                        noted in this email or its attachments, this communication does not form
                                        a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                        **************************************************************************************

                                         

                                         


                                      • Michael Brown
                                        Glad to hear it works for you...I need to get around to playing some more with that stuff. From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com]
                                        Message 19 of 20 , Oct 24, 2011
                                        View Source
                                        • 0 Attachment

                                          Glad to hear it works for you…I need to get around to playing some more with that stuff.

                                           

                                          From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk
                                          Sent: Saturday, October 22, 2011 9:12 AM
                                          To: altdotnet@yahoogroups.com
                                          Subject: Re: [altdotnet] Clustering Azure Queue?

                                           

                                           

                                          I forgot to get back. I tried switching to appfabric service-bus, and that thing is damn fast! I built a one-tenth scale model of my actual production requirement, hitting more than 2000 writes and 2000 reads per seconds, and service-bus hasn't even break a sweat. This thing so far looks pretty invincible. I'm quite happy with it, and it's a lot a lot a lot more cost effective as well, just a fraction of what I would have paid to poll hundreds of queue storage accounts.
                                          Thanks to all who've suggested it.

                                          On Fri, Oct 7, 2011 at 6:40 AM, Michael Brown <mbrown@...> wrote:

                                           

                                          I think the general guidance is to use Azure App fabric for high throughput. Have you done any tests against that for performance? Also, you get billed on number of connections rather than per transaction which is also a better model for very high numbers.

                                           

                                          From: altdotnet@yahoogroups.com [mailto:altdotnet@yahoogroups.com] On Behalf Of Hendry Luk
                                          Sent: Wednesday, October 05, 2011 3:18 AM
                                          To: altdotnet@yahoogroups.com


                                          Subject: Re: [altdotnet] Clustering Azure Queue?

                                           

                                           

                                          Thanks again for the link, it definitely helped me on the right direction, especially Azure Throughput Analyzer from Microsoft Research, which gave me a hard evidence that AzureQueue is capable to enqueue at 750tps when I ran it against my account.
                                          Through reflector, I managed to track it down to Nagle algorithm (which the analyzer code disables). When I disable my own NagleAlgorithm, coupled with ramping up my threads (instead of using the default thread-pool configuration), I finally managed to reach the same level of throughput (750tps).
                                          Which is a massive improvement.

                                          It's still spectacularly low for a queue, but at least it surpasses AzureQueue's quoted throughput (500tps).
                                          To reach 10k tps, I'll definitely still need to round-robin across multiple storage accounts. I recently came across a project that does exactly that (http://partitioncloudqueue.codeplex.com/), although I haven't evaluated how it's any different from my own naiive 3 lines round-robin implementation that I'm currently using.

                                          Thanks for all the help

                                          On Tue, Oct 4, 2011 at 6:08 PM, Hendry Luk <hendrymail@...> wrote:

                                          No I had not. Thank you for that.

                                           

                                          On Thu, Sep 29, 2011 at 12:40 PM, <kelly.leahy@...> wrote:

                                           

                                          Hendry, have you looked at AzureScope?

                                           

                                           

                                          Kelly

                                          -----altdotnet@yahoogroups.com wrote: -----

                                          To: altdotnet@yahoogroups.com
                                          From: Hendry Luk <hendrymail@...>
                                          Sent by: altdotnet@yahoogroups.com
                                          Date: 09/28/2011 06:30PM
                                          Subject: Re: [altdotnet] Clustering Azure Queue?



                                           

                                          I ran few more tests with different combinations, and discovered a very peculiar characteristic in the way azurequeue performs.

                                          1. If I change my producer instance to a bigger VM (more cores and memory), the performance increases linearly. This led me to believe that enqueuing a message is a CPU/memory bound operation, even though I'm using azure-queue async API (BeginAddMessage). It shows that Azure-queue itself is more than powerful enough to take many hundreds of messages per seconds, but the producer is not. (Indeed MS claims azure-queue is capable of taking 500 tps, which now becomes believable).
                                          2. That finding was immediately invalidated when I tried to roundrobin across multiple queue accounts, using the same producer instance (with a small VM), the enqueuing performance increases linearly as I add more and more queue-storage-accounts to round-robin to, which indicates that the VM itself is actually more than powerful enough to make many hundreds of messages per seconds, but each single queue-account is too slow to take them. This clearly contradicts finding#1.
                                          3. I suspected, perhaps the bottleneck is the queue-client object (maybe certain locking mechanism inside). So I used the same round-robin technique (as in point#2), but make them all to point to the same queue-storage-account. It does not help the performance. So this possibility is ruled out.

                                          Those 3 findings do not make any sense to me. They seem to contradict each other. It seems that both azure-queue and the VMs are capable enough to make hundreds of tps, but for some weird reason either one of them has to be made inappropriately massive just to get the other one to its maximum potential. Does anyone have any explanation why azure queue performance behaves the way it does?

                                          NB: from the performance counter, CPU utilization never goes higher than about 52%, and memory never drops below 1.2GB (out of 1.75 total)

                                          On Tue, Sep 27, 2011 at 6:26 PM, Hendry Luk <hendrymail@...> wrote:

                                          I've also just demystified the pricing scheme of azure-queue, and apparently you'll pay per each message-polling. During off-peak time, having hundreds of queue accounts to poll mean a lot of unnecessary fees to pay. This definitely makes appfabric service-bus look more attractive alternative (can increase/decrease number of connections on demands). I'll also look at few other options you've mentioned, and find out the most economically viable option.
                                          Thanks for all the suggestions.

                                           

                                          On Tue, Sep 27, 2011 at 6:06 PM, Simone Busoli <simone.busoli@...> wrote:

                                           

                                          I honestly don't know Hendry, but I know RabbitMQ runs on EC2, and I bet ZeroMQ does as well. For purely technical reasons I think you should be evaluating other messaging systems before picking Azure.
                                          I have worked mostly with RabbitMQ, ActiveMQ and RhinoSB, and I would recommend the former. I think CloudFondry offers a hosted service as well.

                                          On Sep 27, 2011 10:01 AM, "Hendry Luk" <hendrymail@...> wrote:
                                          > Thanks Simon. I wonder if any of those MQ solutions works (properly) on
                                          > azure.
                                          >
                                          > On Tue, Sep 27, 2011 at 5:49 PM, Simone Busoli <simone.busoli@...>wrote:
                                          >
                                          >> **
                                          >>
                                          >>
                                          >> Hi Hendry, never worked with Azure, but I have doubts it has been designed
                                          >> with this throughput in mind. Have you considered alternatives like RabbitMQ
                                          >> or ZeroMQ? Those would probably handle your load way better.
                                          >> On Sep 27, 2011 4:03 AM, "Hendry Luk" <hendrymail@...> wrote:
                                          >> > Hi guys,
                                          >> > I'm developing an application with an expected SLA of 10,000 transactions
                                          >> > per second. From load-tests, I discovered azure-queue to be the
                                          >> bottleneck
                                          >> > of the system, where it can only seem to handle a terrifyingly low 60tps.
                                          >> > I've been trying to look into clustering/load-balancing the queue, with
                                          >> no
                                          >> > much joy so far. Has anyone had any experience/advice on the scalability
                                          >> > story around azure-queue? How can I scale it up to achieve anywhere near
                                          >> the
                                          >> > kind of throughput I need?
                                          >> > Thanks before
                                          >>
                                          >>
                                          >>

                                           

                                           



                                          **************************************************************************************
                                          This communication is intended solely for the addressee and is
                                          confidential. If you are not the intended recipient, any disclosure,
                                          copying, distribution or any action taken or omitted to be taken in
                                          reliance on it, is prohibited and may be unlawful. Unless indicated
                                          to the contrary: it does not constitute professional advice or opinions
                                          upon which reliance may be made by the addressee or any other party,
                                          and it should be considered to be a work in progress. Unless otherwise
                                          noted in this email or its attachments, this communication does not form
                                          a Statement of Actuarial Opinion under American Academy of Actuaries guidelines.
                                          **************************************************************************************

                                           

                                           

                                           

                                        Your message has been successfully submitted and would be delivered to recipients shortly.