Loading ...
Sorry, an error occurred while loading the content.

Handling failures

Expand Messages
  • pawel.pabich
    As far as I know if a message delivery fails then NServiceBus retries immediately and keeps doing this until 5th attempt in a row fails. If that happens then
    Message 1 of 20 , May 20, 2008
    View Source
    • 0 Attachment
      As far as I know if a message delivery fails then NServiceBus retries
      immediately and keeps doing this until 5th attempt in a row fails. If
      that happens then it sends the message to the error queue. Is there
      any setting that would instruct NServiceBus to wait some period of
      time between retries? Is there any other built-in way of handling such
      cases when the first delivery fails?

      thanks

      Pawel
    • Udi Dahan
      The retry count doesn t have to do with message delivery. MSMQ handles that independently of nServiceBus. When a transaction is thrown during any part of
      Message 2 of 20 , May 20, 2008
      View Source
      • 0 Attachment

        The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

         

        When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

         

        The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

         

        Does that answer your question?

         

        --

        Udi Dahan - The Software Simplist

         

        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
        Sent: 21 May 2008 00:48
        To: nservicebus@yahoogroups.com
        Subject: [nservicebus] Handling failures

         

        As far as I know if a message delivery fails then NServiceBus retries
        immediately and keeps doing this until 5th attempt in a row fails. If
        that happens then it sends the message to the error queue. Is there
        any setting that would instruct NServiceBus to wait some period of
        time between retries? Is there any other built-in way of handling such
        cases when the first delivery fails?

        thanks

        Pawel

        No virus found in this incoming message.
        Checked by AVG.
        Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

      • Andreas Öhlund
        Isn t Pawel asking about some kind of Retry strategy . The current one is Retry 5 time with no wait . I ve seen solutions that sleeps a while before retrying
        Message 3 of 20 , May 20, 2008
        View Source
        • 0 Attachment
          Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
           
          It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
           
          Thoughts?





          To: nservicebus@yahoogroups.com
          From: thesoftwaresimplist@...
          Date: Wed, 21 May 2008 01:27:43 +0300
          Subject: RE: [nservicebus] Handling failures

          The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

           

          When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

           

          The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

           

          Does that answer your question?

           

          --

          Udi Dahan - The Software Simplist

           

          From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
          Sent: 21 May 2008 00:48
          To: nservicebus@yahoogroups.com
          Subject: [nservicebus] Handling failures

           

          As far as I know if a message delivery fails then NServiceBus retries
          immediately and keeps doing this until 5th attempt in a row fails. If
          that happens then it sends the message to the error queue. Is there
          any setting that would instruct NServiceBus to wait some period of
          time between retries? Is there any other built-in way of handling such
          cases when the first delivery fails?

          thanks

          Pawel

          No virus found in this incoming message.
          Checked by AVG.
          Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45



          Den perfekta mixen av nöjesnyheter & livekonserter! MSN Video
        • Udi Dahan
          My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself. That’s one example as to a case
          Message 4 of 20 , May 21, 2008
          View Source
          • 0 Attachment

            My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

             

            That’s one example as to a case where a different transport implementation would expose a different extensibility model.

             

            --

            Udi Dahan - The Software Simplist

             

            From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
            Sent: 21 May 2008 09:59
            To: nservicebus@yahoogroups.com
            Subject: RE: [nservicebus] Handling failures

             

            Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
             
            It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
             
            Thoughts?




            To: nservicebus@yahoogroups.com
            From: thesoftwaresimplist@...
            Date: Wed, 21 May 2008 01:27:43 +0300
            Subject: RE: [nservicebus] Handling failures

            The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

             

            When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

             

            The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

             

            Does that answer your question?

             

            --

            Udi Dahan - The Software Simplist

             

            From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
            Sent: 21 May 2008 00:48
            To: nservicebus@yahoogroups.com
            Subject: [nservicebus] Handling failures

             

            As far as I know if a message delivery fails then NServiceBus retries
            immediately and keeps doing this until 5th attempt in a row fails. If
            that happens then it sends the message to the error queue. Is there
            any setting that would instruct NServiceBus to wait some period of
            time between retries? Is there any other built-in way of handling such
            cases when the first delivery fails?

            thanks

            Pawel

            No virus found in this incoming message.
            Checked by AVG.
            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

             


            Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

            No virus found in this incoming message.
            Checked by AVG.
            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

          • pawel
            Hi, If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps. 1. The message is sent to the sender local queue. 2.
            Message 5 of 20 , May 21, 2008
            View Source
            • 0 Attachment
              Hi,

              If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

              1. The message is sent to the sender local queue.
              2. The message is transported from sender local queue to recipient local queue.
              3. The message is retrieved from the recipient local queue and the recipient processes the message.

              The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
              5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
              support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
              that deals with failures when the message is actually being processed by the recipient.

              thanks

              Pawel


              From: "Udi Dahan" <thesoftwaresimplist@...>
              Sent: 21 May 2008 08:53
              To: nservicebus@yahoogroups.com
              Subject: RE: [nservicebus] Handling failures


              My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

               

              That’s one example as to a case where a different transport implementation would expose a different extensibility model.

               

              --

              Udi Dahan - The Software Simplist

               

              From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of Andreas Öhlund
              Sent: 21 May 2008 09:59
              To: nservicebus@ yahoogroups. com
              Subject: RE: [nservicebus] Handling failures

               

              Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
               
              It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFail edEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
               
              Thoughts?




              To: nservicebus@ yahoogroups. com
              From: thesoftwaresimplist @...
              Date: Wed, 21 May 2008 01:27:43 +0300
              Subject: RE: [nservicebus] Handling failures

              The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

               

              When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

               

              The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

               

              Does that answer your question?

               

              --

              Udi Dahan - The Software Simplist

               

              From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel.pabich
              Sent: 21 May 2008 00:48
              To: nservicebus@ yahoogroups. com
              Subject: [nservicebus] Handling failures

               

              As far as I know if a message delivery fails then NServiceBus retries
              immediately and keeps doing this until 5th attempt in a row fails. If
              that happens then it sends the message to the error queue. Is there
              any setting that would instruct NServiceBus to wait some period of
              time between retries? Is there any other built-in way of handling such
              cases when the first delivery fails?

              thanks

              Pawel

              No virus found in this incoming message.
              Checked by AVG.
              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

               


              Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

              No virus found in this incoming message.
              Checked by AVG.
              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45


            • Udi Dahan
              Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to
              Message 6 of 20 , May 21, 2008
              View Source
              • 0 Attachment

                Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                 

                When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                 

                That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                 

                While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                 

                Hope that explains the reasoning a bit.

                 

                --

                Udi Dahan - The Software Simplist

                 

                From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                Sent: 21 May 2008 13:16
                To: nservicebus@yahoogroups.com
                Subject: RE: [nservicebus] Handling failures

                 

                Hi,

                If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                1. The message is sent to the sender local queue.
                2. The message is transported from sender local queue to recipient local queue.
                3. The message is retrieved from the recipient local queue and the recipient processes the message.

                The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                that deals with failures when the message is actually being processed by the recipient.

                thanks

                Pawel


                From: "Udi Dahan" <thesoftwaresimplist@...>
                Sent: 21 May 2008 08:53
                To: nservicebus@yahoogroups.com
                Subject: RE: [nservicebus] Handling failures

                My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                 

                That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                 

                --

                Udi Dahan - The Software Simplist

                 

                From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                Sent: 21 May 2008 09:59
                To: nservicebus@yahoogroups.com
                Subject: RE: [nservicebus] Handling failures

                 

                Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                 
                It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                 
                Thoughts?





                To: nservicebus@yahoogroups.com
                From: thesoftwaresimplist@...
                Date: Wed, 21 May 2008 01:27:43 +0300
                Subject: RE: [nservicebus] Handling failures

                The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                 

                When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                 

                The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                 

                Does that answer your question?

                 

                --

                Udi Dahan - The Software Simplist

                 

                From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                Sent: 21 May 2008 00:48
                To: nservicebus@yahoogroups.com
                Subject: [nservicebus] Handling failures

                 

                As far as I know if a message delivery fails then NServiceBus retries
                immediately and keeps doing this until 5th attempt in a row fails. If
                that happens then it sends the message to the error queue. Is there
                any setting that would instruct NServiceBus to wait some period of
                time between retries? Is there any other built-in way of handling such
                cases when the first delivery fails?

                thanks

                Pawel

                No virus found in this incoming message.
                Checked by AVG.
                Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                 


                Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                No virus found in this incoming message.
                Checked by AVG.
                Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                 

                No virus found in this incoming message.
                Checked by AVG.
                Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

              • pawel
                I see your point and it makes sense as long as you control external systems. If you don t and they are a bit flaky than a good retry strategy might help a lot.
                Message 7 of 20 , May 21, 2008
                View Source
                • 0 Attachment
                  I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                  retry strategy might help a lot.

                  thanks again

                  Pawel


                  From: "Udi Dahan" <thesoftwaresimplist@...>
                  Sent: 21 May 2008 13:12
                  To: nservicebus@yahoogroups.com
                  Subject: RE: [nservicebus] Handling failures


                  Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                   

                  When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                   

                  That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                   

                  While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                   

                  Hope that explains the reasoning a bit.

                   

                  --

                  Udi Dahan - The Software Simplist

                   

                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                  Sent: 21 May 2008 13:16
                  To: nservicebus@ yahoogroups.. com
                  Subject: RE: [nservicebus] Handling failures

                   

                  Hi,

                  If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                  1. The message is sent to the sender local queue.
                  2. The message is transported from sender local queue to recipient local queue.
                  3. The message is retrieved from the recipient local queue and the recipient processes the message.

                  The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                  5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                  support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                  that deals with failures when the message is actually being processed by the recipient.

                  thanks

                  Pawel


                  From: "Udi Dahan" <thesoftwaresimplist @...>
                  Sent: 21 May 2008 08:53
                  To: nservicebus@ yahoogroups. com
                  Subject: RE: [nservicebus] Handling failures

                  My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                   

                  That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                   

                  --

                  Udi Dahan - The Software Simplist

                   

                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of Andreas Öhlund
                  Sent: 21 May 2008 09:59
                  To: nservicebus@ yahoogroups. com
                  Subject: RE: [nservicebus] Handling failures

                   

                  Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                   
                  It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFail edEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                   
                  Thoughts?





                  To: nservicebus@ yahoogroups. com
                  From: thesoftwaresimplist @...
                  Date: Wed, 21 May 2008 01:27:43 +0300
                  Subject: RE: [nservicebus] Handling failures

                  The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                   

                  When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                   

                  The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                   

                  Does that answer your question?

                   

                  --

                  Udi Dahan - The Software Simplist

                   

                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel.pabich
                  Sent: 21 May 2008 00:48
                  To: nservicebus@ yahoogroups. com
                  Subject: [nservicebus] Handling failures

                   

                  As far as I know if a message delivery fails then NServiceBus retries
                  immediately and keeps doing this until 5th attempt in a row fails. If
                  that happens then it sends the message to the error queue. Is there
                  any setting that would instruct NServiceBus to wait some period of
                  time between retries? Is there any other built-in way of handling such
                  cases when the first delivery fails?

                  thanks

                  Pawel

                  No virus found in this incoming message.
                  Checked by AVG.
                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                   


                  Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                  No virus found in this incoming message.
                  Checked by AVG.
                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                   

                  No virus found in this incoming message.
                  Checked by AVG.
                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45


                • Udi Dahan
                  In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion? --
                  Message 8 of 20 , May 21, 2008
                  View Source
                  • 0 Attachment

                    In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion?

                     

                    --

                    Udi Dahan - The Software Simplist

                     

                    From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                    Sent: 21 May 2008 16:14
                    To: nservicebus@yahoogroups.com
                    Subject: RE: [nservicebus] Handling failures

                     

                    I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                    retry strategy might help a lot.

                    thanks again

                    Pawel


                    From: "Udi Dahan" <thesoftwaresimplist@...>
                    Sent: 21 May 2008 13:12
                    To: nservicebus@yahoogroups.com
                    Subject: RE: [nservicebus] Handling failures

                    Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                     

                    When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                     

                    That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                     

                    While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                     

                    Hope that explains the reasoning a bit.

                     

                    --

                    Udi Dahan - The Software Simplist

                     

                    From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                    Sent: 21 May 2008 13:16
                    To: nservicebus@yahoogroups..com
                    Subject: RE: [nservicebus] Handling failures

                     

                    Hi,

                    If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                    1. The message is sent to the sender local queue.
                    2. The message is transported from sender local queue to recipient local queue.
                    3. The message is retrieved from the recipient local queue and the recipient processes the message.

                    The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                    5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                    support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                    that deals with failures when the message is actually being processed by the recipient.

                    thanks

                    Pawel


                    From: "Udi Dahan" <thesoftwaresimplist@...>
                    Sent: 21 May 2008 08:53
                    To: nservicebus@yahoogroups.com
                    Subject: RE: [nservicebus] Handling failures

                    My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                     

                    That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                     

                    --

                    Udi Dahan - The Software Simplist

                     

                    From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                    Sent: 21 May 2008 09:59
                    To: nservicebus@yahoogroups.com
                    Subject: RE: [nservicebus] Handling failures

                     

                    Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                     
                    It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                     
                    Thoughts?






                    To: nservicebus@yahoogroups.com
                    From: thesoftwaresimplist@...
                    Date: Wed, 21 May 2008 01:27:43 +0300
                    Subject: RE: [nservicebus] Handling failures

                    The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                     

                    When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                     

                    The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                     

                    Does that answer your question?

                     

                    --

                    Udi Dahan - The Software Simplist

                     

                    From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                    Sent: 21 May 2008 00:48
                    To: nservicebus@yahoogroups.com
                    Subject: [nservicebus] Handling failures

                     

                    As far as I know if a message delivery fails then NServiceBus retries
                    immediately and keeps doing this until 5th attempt in a row fails. If
                    that happens then it sends the message to the error queue. Is there
                    any setting that would instruct NServiceBus to wait some period of
                    time between retries? Is there any other built-in way of handling such
                    cases when the first delivery fails?

                    thanks

                    Pawel

                    No virus found in this incoming message.
                    Checked by AVG.
                    Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                     


                    Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                    No virus found in this incoming message.
                    Checked by AVG.
                    Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                     

                    No virus found in this incoming message.
                    Checked by AVG.
                    Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                     

                    No virus found in this incoming message.
                    Checked by AVG.
                    Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                  • Eric Hauser
                    I don t understand why retry handling would need to be part of the framework? If you wanted to do that, couldn t you just do something like (prototype code):
                    Message 9 of 20 , May 21, 2008
                    View Source
                    • 0 Attachment
                      I don't understand why retry handling would need to be part of the
                      framework? If you wanted to do that, couldn't you just do something
                      like (prototype code):


                      public interface IReplayableMessage : IMessage
                      {
                      int RetryCount { get; set; }
                      DateTime LastReplayAttempt { get; set; }
                      }

                      public abstract class RetryMessageHandler<TMessage> :
                      IMessageHandler<TMessage>
                      where TMessage : IReplayableMessage
                      {
                      public IBus Bus { get; set; }

                      public void HandleMessage(TMessage message)
                      {
                      if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                      {
                      Bus.HandleMessageLater();
                      return;
                      }
                      try
                      {
                      HandleMessageInternal(message);
                      }
                      catch (Exception e)
                      {
                      if (message.RetryCount < 5)
                      {
                      message.RetryCount = message.RetryCount++;
                      message.LastReplayAttempt = DateTime.Now;
                      Bus.Send(message);
                      }
                      else
                      {
                      .. log error ..
                      }
                      }
                      }

                      public abstract void HandleMessageInternal(TMessage message)
                      }



                      --- In nservicebus@yahoogroups.com, "Udi Dahan"
                      <thesoftwaresimplist@...> wrote:
                      >
                      > In the example I was talking about, what do you view as the
                      external system? Or does the fact that I assume MSMQ communication
                      lead you to that conclusion?
                      >
                      >
                      >
                      > --
                      >
                      > Udi Dahan - The Software Simplist
                      >
                      >
                      >
                      > From: nservicebus@yahoogroups.com
                      [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                      > Sent: 21 May 2008 16:14
                      > To: nservicebus@yahoogroups.com
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      >
                      >
                      > I see your point and it makes sense as long as you control external
                      systems. If you don't and they are a bit flaky than a good
                      > retry strategy might help a lot.
                      >
                      > thanks again
                      >
                      > Pawel
                      >
                      > _____
                      >
                      > From: "Udi Dahan" <thesoftwaresimplist@...>
                      > Sent: 21 May 2008 13:12
                      > To: nservicebus@yahoogroups.com
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      > Actually, when you send a message, it first goes to the outgoing
                      queue on the sending machine. Once that successfully occurs, the
                      sending thread is released to go about its business. Then MSMQ tries
                      to get the message to the specified queue on the target machine. Once
                      that occurs, the message is deleted from the outgoing queue of the
                      sending machine. From that point on, the message is sitting in a
                      queue waiting to be processed. When the target process retrieves the
                      message from that queue (its input queue), it processes it.
                      >
                      >
                      >
                      > When the DB detects a deadlock between two threads, it kicks one
                      out, and lets the other proceed. Given that transactions are limited
                      in scope to a single message, it is rare that the same thread will
                      hit a deadlock on the same data when it tries N micro/milliseconds
                      later, and, even if so, half that chance that it will be selected as
                      the victim again. The N micros/millis is the time it takes to
                      rollback the transaction and try again.
                      >
                      >
                      >
                      > That being said, you should be setting the appropriate number of
                      threads in your autonomous components so that you get the maximal
                      throughput while maintaining minimal contention. The more threads are
                      NOT the better. Also, poor DB design can increase the chance of
                      contention, as can processing large messages where transactions are
                      open longer. This is all applicative design, and has little to do
                      with a retry strategy.
                      >
                      >
                      >
                      > While I can’t categorically say that a retry strategy isn’t
                      needed, I can say that I’ve handled very, VERY heavy loads without
                      needing one, just by profiling, setting the appropriate number of
                      threads, and “just” by decreasing message size (as my InfoQ
                      article explained).
                      >
                      >
                      >
                      > Hope that explains the reasoning a bit.
                      >
                      >
                      >
                      > --
                      >
                      > Udi Dahan - The Software Simplist
                      >
                      >
                      >
                      > From: nservicebus@yahoogroups.com
                      [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                      > Sent: 21 May 2008 13:16
                      > To: nservicebus@...
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      >
                      >
                      > Hi,
                      >
                      > If I understand correctly then whenever a message is sent using
                      MSMQ we deal with 3 distinct steps.
                      >
                      > 1. The message is sent to the sender local queue.
                      > 2. The message is transported from sender local queue to recipient
                      local queue.
                      > 3. The message is retrieved from the recipient local queue and the
                      recipient processes the message.
                      >
                      > The re-try strategy that I'm talking about is related to the 3rd
                      step. If a database is under heavy load then re-trying
                      > 5 times in a row (without waiting between attempts) is not going to
                      help and MSMQ itself does not provide any
                      > support at this stage(except transactional processing). I was
                      wondering if there is a more sophisticated built-in strategy
                      > that deals with failures when the message is actually being
                      processed by the recipient.
                      >
                      > thanks
                      >
                      > Pawel
                      >
                      > _____
                      >
                      > From: "Udi Dahan" <thesoftwaresimplist@...>
                      > Sent: 21 May 2008 08:53
                      > To: nservicebus@yahoogroups.com
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      > My answer was to say that nServiceBus currently doesn’t do those
                      kinds of retry strategies â€" MSMQ does that by itself.
                      >
                      >
                      >
                      > That’s one example as to a case where a different transport
                      implementation would expose a different extensibility model.
                      >
                      >
                      >
                      > --
                      >
                      > Udi Dahan - The Software Simplist
                      >
                      >
                      >
                      > From: nservicebus@yahoogroups.com
                      [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                      > Sent: 21 May 2008 09:59
                      > To: nservicebus@yahoogroups.com
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      >
                      >
                      > Isn't Pawel asking about some kind of "Retry strategy". The current
                      one is "Retry 5 time with no wait". I've seen solutions that sleeps a
                      while before retrying ( and perhaps extends the "sleep period" based
                      on number of failed retries)
                      >
                      > It would also be nice if the bus or transport could expose
                      an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know
                      you could read the error queue but this would be a more transport
                      agnostic solution?)
                      >
                      > Thoughts?
                      >
                      >
                      >
                      >
                      >
                      >
                      >
                      > _____
                      >
                      > To: nservicebus@yahoogroups.com
                      > From: thesoftwaresimplist@...
                      > Date: Wed, 21 May 2008 01:27:43 +0300
                      > Subject: RE: [nservicebus] Handling failures
                      >
                      > The retry count doesn’t have to do with message delivery. MSMQ
                      handles that independently of nServiceBus.
                      >
                      >
                      >
                      > When a transaction is thrown during any part of processing the
                      message, then we retry the HANDLING of the message. Deserialization
                      exceptions are different â€" those are immediately transferred to the
                      error queue without any retries.
                      >
                      >
                      >
                      > The reason for this logic has to do with technological issues
                      preventing otherwise successful message handling â€" deadlocks in the
                      database are one example.
                      >
                      >
                      >
                      > Does that answer your question?
                      >
                      >
                      >
                      > --
                      >
                      > Udi Dahan - The Software Simplist
                      >
                      >
                      >
                      > From: nservicebus@yahoogroups.com
                      [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                      > Sent: 21 May 2008 00:48
                      > To: nservicebus@yahoogroups.com
                      > Subject: [nservicebus] Handling failures
                      >
                      >
                      >
                      > As far as I know if a message delivery fails then NServiceBus
                      retries
                      > immediately and keeps doing this until 5th attempt in a row fails.
                      If
                      > that happens then it sends the message to the error queue. Is there
                      > any setting that would instruct NServiceBus to wait some period of
                      > time between retries? Is there any other built-in way of handling
                      such
                      > cases when the first delivery fails?
                      >
                      > thanks
                      >
                      > Pawel
                      >
                      > No virus found in this incoming message.
                      > Checked by AVG.
                      > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                      20/05/2008 16:45
                      >
                      >
                      >
                      > _____
                      >
                      > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                      <http://video.se.msn.com/v/sv-se/v.htm>
                      >
                      > No virus found in this incoming message.
                      > Checked by AVG.
                      > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                      20/05/2008 16:45
                      >
                      >
                      >
                      > No virus found in this incoming message.
                      > Checked by AVG.
                      > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                      20/05/2008 16:45
                      >
                      >
                      >
                      >
                      >
                      > No virus found in this incoming message.
                      > Checked by AVG.
                      > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                      20/05/2008 16:45
                      >
                    • Udi Dahan
                      That’s an interesting thought – using nServiceBus under the ITransport level to implement that functionality. It would break configuration based on type,
                      Message 10 of 20 , May 22, 2008
                      View Source
                      • 0 Attachment

                        That’s an interesting thought – using nServiceBus under the ITransport level to implement that functionality. It would break configuration based on type, though.

                         

                        --

                        Udi Dahan - The Software Simplist

                         

                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                        Sent: 22 May 2008 03:52
                        To: nservicebus@yahoogroups.com
                        Subject: [nservicebus] Re: Handling failures

                         

                        I don't understand why retry handling would need to be part of the
                        framework? If you wanted to do that, couldn't you just do something
                        like (prototype code):

                        public interface IReplayableMessage : IMessage
                        {
                        int RetryCount { get; set; }
                        DateTime LastReplayAttempt { get; set; }
                        }

                        public abstract class RetryMessageHandler<TMessage> :
                        IMessageHandler<TMessage>
                        where TMessage : IReplayableMessage
                        {
                        public IBus Bus { get; set; }

                        public void HandleMessage(TMessage message)
                        {
                        if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                        {
                        Bus.HandleMessageLater();
                        return;
                        }
                        try
                        {
                        HandleMessageInternal(message);
                        }
                        catch (Exception e)
                        {
                        if (message.RetryCount < 5)
                        {
                        message.RetryCount = message.RetryCount++;
                        message.LastReplayAttempt = DateTime.Now;
                        Bus.Send(message);
                        }
                        else
                        {
                        .. log error ..
                        }
                        }
                        }

                        public abstract void HandleMessageInternal(TMessage message)
                        }

                        --- In nservicebus@yahoogroups.com, "Udi Dahan"
                        <thesoftwaresimplist@...> wrote:

                        >
                        > In the example I was talking about, what do you view as the
                        external system? Or does the fact that I assume MSMQ communication
                        lead you to that conclusion?
                        >
                        >
                        >
                        > --
                        >
                        > Udi Dahan - The Software Simplist
                        >
                        >
                        >
                        > From: nservicebus@yahoogroups.com
                        [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                        > Sent: 21 May 2008 16:14
                        > To: nservicebus@yahoogroups.com
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        >
                        >
                        > I see your point and it makes sense as long as you control external
                        systems. If you don't and they are a bit flaky than a good
                        > retry strategy might help a lot.
                        >
                        > thanks again
                        >
                        > Pawel
                        >
                        > _____
                        >
                        > From: "Udi Dahan" <thesoftwaresimplist@...>
                        > Sent: 21 May 2008 13:12
                        > To: nservicebus@yahoogroups.com
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        > Actually, when you send a message, it first goes to the outgoing
                        queue on the sending machine. Once that successfully occurs, the
                        sending thread is released to go about its business. Then MSMQ tries
                        to get the message to the specified queue on the target machine. Once
                        that occurs, the message is deleted from the outgoing queue of the
                        sending machine. From that point on, the message is sitting in a
                        queue waiting to be processed. When the target process retrieves the
                        message from that queue (its input queue), it processes it.
                        >
                        >
                        >
                        > When the DB detects a deadlock between two threads, it kicks one
                        out, and lets the other proceed. Given that transactions are limited
                        in scope to a single message, it is rare that the same thread will
                        hit a deadlock on the same data when it tries N micro/milliseconds
                        later, and, even if so, half that chance that it will be selected as
                        the victim again. The N micros/millis is the time it takes to
                        rollback the transaction and try again.
                        >
                        >
                        >
                        > That being said, you should be setting the appropriate number of
                        threads in your autonomous components so that you get the maximal
                        throughput while maintaining minimal contention. The more threads are
                        NOT the better. Also, poor DB design can increase the chance of
                        contention, as can processing large messages where transactions are
                        open longer. This is all applicative design, and has little to do
                        with a retry strategy.
                        >
                        >
                        >
                        > While I can’t categorically say that a retry strategy isn’t
                        needed, I can say that I’ve handled very, VERY heavy loads without
                        needing one, just by profiling, setting the appropriate number of
                        threads, and “just” by decreasing message size (as my InfoQ
                        article explained).
                        >
                        >
                        >
                        > Hope that explains the reasoning a bit.
                        >
                        >
                        >
                        > --
                        >
                        > Udi Dahan - The Software Simplist
                        >
                        >
                        >
                        > From: nservicebus@yahoogroups.com
                        [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                        > Sent: 21 May 2008 13:16
                        > To: nservicebus@...
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        >
                        >
                        > Hi,
                        >
                        > If I understand correctly then whenever a message is sent using
                        MSMQ we deal with 3 distinct steps.
                        >
                        > 1. The message is sent to the sender local queue.
                        > 2. The message is transported from sender local queue to recipient
                        local queue.
                        > 3. The message is retrieved from the recipient local queue and the
                        recipient processes the message.
                        >
                        > The re-try strategy that I'm talking about is related to the 3rd
                        step. If a database is under heavy load then re-trying
                        > 5 times in a row (without waiting between attempts) is not going to
                        help and MSMQ itself does not provide any
                        > support at this stage(except transactional processing). I was
                        wondering if there is a more sophisticated built-in strategy
                        > that deals with failures when the message is actually being
                        processed by the recipient.
                        >
                        > thanks
                        >
                        > Pawel
                        >
                        > _____
                        >
                        > From: "Udi Dahan" <thesoftwaresimplist@...>
                        > Sent: 21 May 2008 08:53
                        > To: nservicebus@yahoogroups.com
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        > My answer was to say that nServiceBus currently doesn’t do those
                        kinds of retry strategies â€" MSMQ does that by itself.
                        >
                        >
                        >
                        > That’s one example as to a case where a different transport
                        implementation would expose a different extensibility model.
                        >
                        >
                        >
                        > --
                        >
                        > Udi Dahan - The Software Simplist
                        >
                        >
                        >
                        > From: nservicebus@yahoogroups.com
                        [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                        > Sent: 21 May 2008 09:59
                        > To: nservicebus@yahoogroups.com
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        >
                        >
                        > Isn't Pawel asking about some kind of "Retry strategy". The
                        current
                        one is "Retry 5 time with no wait". I've seen solutions that sleeps a
                        while before retrying ( and perhaps extends the "sleep period" based
                        on number of failed retries)
                        >
                        > It would also be nice if the bus or transport could expose
                        an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know
                        you could read the error queue but this would be a more transport
                        agnostic solution?)
                        >
                        > Thoughts?
                        >
                        >
                        >
                        >
                        >
                        >
                        >
                        > _____
                        >
                        > To: nservicebus@yahoogroups.com
                        > From: thesoftwaresimplist@...
                        > Date: Wed, 21 May 2008 01:27:43 +0300
                        > Subject: RE: [nservicebus] Handling failures
                        >
                        > The retry count doesn’t have to do with message delivery. MSMQ
                        handles that independently of nServiceBus.
                        >
                        >
                        >
                        > When a transaction is thrown during any part of processing the
                        message, then we retry the HANDLING of the message. Deserialization
                        exceptions are different â€" those are immediately transferred to the
                        error queue without any retries.
                        >
                        >
                        >
                        > The reason for this logic has to do with technological issues
                        preventing otherwise successful message handling â€" deadlocks in the
                        database are one example.
                        >
                        >
                        >
                        > Does that answer your question?
                        >
                        >
                        >
                        > --
                        >
                        > Udi Dahan - The Software Simplist
                        >
                        >
                        >
                        > From: nservicebus@yahoogroups.com
                        [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                        > Sent: 21 May 2008 00:48
                        > To: nservicebus@yahoogroups.com
                        > Subject: [nservicebus] Handling failures
                        >
                        >
                        >
                        > As far as I know if a message delivery fails then NServiceBus
                        retries
                        > immediately and keeps doing this until 5th attempt in a row fails.
                        If
                        > that happens then it sends the message to the error queue. Is there
                        > any setting that would instruct NServiceBus to wait some period of
                        > time between retries? Is there any other built-in way of handling
                        such
                        > cases when the first delivery fails?
                        >
                        > thanks
                        >
                        > Pawel
                        >
                        > No virus found in this incoming message.
                        > Checked by AVG.
                        > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                        20/05/2008 16:45
                        >
                        >
                        >
                        > _____
                        >
                        > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                        <http://video.se.msn.com/v/sv-se/v.htm>
                        >
                        > No virus found in this incoming message.
                        > Checked by AVG.
                        > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                        20/05/2008 16:45
                        >
                        >
                        >
                        > No virus found in this incoming message.
                        > Checked by AVG.
                        > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                        20/05/2008 16:45
                        >
                        >
                        >
                        >
                        >
                        > No virus found in this incoming message.
                        > Checked by AVG.
                        > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                        20/05/2008 16:45
                        >

                        No virus found in this incoming message.
                        Checked by AVG.
                        Version: 8.0.100 / Virus Database: 269.23.21/1458 - Release Date: 21/05/2008 07:21

                      • pawel
                        There is always this question where you draw the line between application code and infrastructure code. In this particular case I would see the retry strategy
                        Message 11 of 20 , May 22, 2008
                        View Source
                        • 0 Attachment
                          There is always this question where you draw the line between application code and infrastructure code.
                          In this particular case I would see the retry strategy at the infrastructure level. The application code decides only
                          whether it's able to process a message or not. I think it's more a matter of taste which way you want to go.
                          Anyway it looks like your solution can be implemented and applied quickly which is great.

                          thanks

                          Pawel


                          From: "Eric Hauser" <ewhauser@...>
                          Sent: 22 May 2008 02:19
                          To: nservicebus@yahoogroups.com
                          Subject: [nservicebus] Re: Handling failures


                          I don't understand why retry handling would need to be part of the
                          framework? If you wanted to do that, couldn't you just do something
                          like (prototype code):

                          public interface IReplayableMessage : IMessage
                          {
                          int RetryCount { get; set; }
                          DateTime LastReplayAttempt { get; set; }
                          }

                          public abstract class RetryMessageHandler <TMessage> :
                          IMessageHandler< TMessage>
                          where TMessage : IReplayableMessage
                          {
                          public IBus Bus { get; set; }

                          public void HandleMessage( TMessage message)
                          {
                          if (DateTime.Now - message.LastReplayA ttempt > 5.Seconds())
                          {
                          Bus.HandleMessageLa ter();
                          return;
                          }
                          try
                          {
                          HandleMessageIntern al(message) ;
                          }
                          catch (Exception e)
                          {
                          if (message.RetryCount < 5)
                          {
                          message.RetryCount = message.RetryCount+ +;
                          message.LastReplayA ttempt = DateTime.Now;
                          Bus.Send(message) ;
                          }
                          else
                          {
                          .. log error ..
                          }
                          }
                          }

                          public abstract void HandleMessageIntern al(TMessage message)
                          }

                          --- In nservicebus@ yahoogroups. com, "Udi Dahan"
                          <thesoftwaresimplis t@...> wrote:

                          >
                          > In the example I was talking about, what do you view as the
                          external system? Or does the fact that I assume MSMQ communication
                          lead you to that conclusion?
                          >
                          >
                          >
                          > --
                          >
                          > Udi Dahan - The Software Simplist
                          >
                          >
                          >
                          > From: nservicebus@ yahoogroups. com
                          [mailto:nservicebus@ yahoogroups. com] On Behalf Of pawel
                          > Sent: 21 May 2008 16:14
                          > To: nservicebus@ yahoogroups. com
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          >
                          >
                          > I see your point and it makes sense as long as you control external
                          systems. If you don't and they are a bit flaky than a good
                          > retry strategy might help a lot.
                          >
                          > thanks again
                          >
                          > Pawel
                          >
                          > _____
                          >
                          > From: "Udi Dahan" <thesoftwaresimplis t@...>
                          > Sent: 21 May 2008 13:12
                          > To: nservicebus@ yahoogroups. com
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          > Actually, when you send a message, it first goes to the outgoing
                          queue on the sending machine. Once that successfully occurs, the
                          sending thread is released to go about its business. Then MSMQ tries
                          to get the message to the specified queue on the target machine. Once
                          that occurs, the message is deleted from the outgoing queue of the
                          sending machine. From that point on, the message is sitting in a
                          queue waiting to be processed. When the target process retrieves the
                          message from that queue (its input queue), it processes it.
                          >
                          >
                          >
                          > When the DB detects a deadlock between two threads, it kicks one
                          out, and lets the other proceed. Given that transactions are limited
                          in scope to a single message, it is rare that the same thread will
                          hit a deadlock on the same data when it tries N micro/milliseconds
                          later, and, even if so, half that chance that it will be selected as
                          the victim again. The N micros/millis is the time it takes to
                          rollback the transaction and try again.
                          >
                          >
                          >
                          > That being said, you should be setting the appropriate number of
                          threads in your autonomous components so that you get the maximal
                          throughput while maintaining minimal contention. The more threads are
                          NOT the better. Also, poor DB design can increase the chance of
                          contention, as can processing large messages where transactions are
                          open longer. This is all applicative design, and has little to do
                          with a retry strategy.
                          >
                          >
                          >
                          > While I can’t categorically say that a retry strategy isn’t
                          needed, I can say that I’ve handled very, VERY heavy loads without
                          needing one, just by profiling, setting the appropriate number of
                          threads, and “just” by decreasing message size (as my InfoQ
                          article explained).
                          >
                          >
                          >
                          > Hope that explains the reasoning a bit.
                          >
                          >
                          >
                          > --
                          >
                          > Udi Dahan - The Software Simplist
                          >
                          >
                          >
                          > From: nservicebus@ yahoogroups. com
                          [mailto:nservicebus@ yahoogroups. com] On Behalf Of pawel
                          > Sent: 21 May 2008 13:16
                          > To: nservicebus@ ...
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          >
                          >
                          > Hi,
                          >
                          > If I understand correctly then whenever a message is sent using
                          MSMQ we deal with 3 distinct steps.
                          >
                          > 1. The message is sent to the sender local queue.
                          > 2. The message is transported from sender local queue to recipient
                          local queue.
                          > 3. The message is retrieved from the recipient local queue and the
                          recipient processes the message.
                          >
                          > The re-try strategy that I'm talking about is related to the 3rd
                          step. If a database is under heavy load then re-trying
                          > 5 times in a row (without waiting between attempts) is not going to
                          help and MSMQ itself does not provide any
                          > support at this stage(except transactional processing). I was
                          wondering if there is a more sophisticated built-in strategy
                          > that deals with failures when the message is actually being
                          processed by the recipient.
                          >
                          > thanks
                          >
                          > Pawel
                          >
                          > _____
                          >
                          > From: "Udi Dahan" <thesoftwaresimplis t@...>
                          > Sent: 21 May 2008 08:53
                          > To: nservicebus@ yahoogroups. com
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          > My answer was to say that nServiceBus currently doesn’t do those
                          kinds of retry strategies â€" MSMQ does that by itself.
                          >
                          >
                          >
                          > That’s one example as to a case where a different transport
                          implementation would expose a different extensibility model.
                          >
                          >
                          >
                          > --
                          >
                          > Udi Dahan - The Software Simplist
                          >
                          >
                          >
                          > From: nservicebus@ yahoogroups. com
                          [mailto:nservicebus@ yahoogroups. com] On Behalf Of Andreas Öhlund
                          > Sent: 21 May 2008 09:59
                          > To: nservicebus@ yahoogroups. com
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          >
                          >
                          > Isn't Pawel asking about some kind of "Retry strategy". The current
                          one is "Retry 5 time with no wait". I've seen solutions that sleeps a
                          while before retrying ( and perhaps extends the "sleep period" based
                          on number of failed retries)
                          >
                          > It would also be nice if the bus or transport could expose
                          an "MessageRetriedEven t" and a "MessageHandlingFai ledEvent" (I know
                          you could read the error queue but this would be a more transport
                          agnostic solution?)
                          >
                          > Thoughts?
                          >
                          >
                          >
                          >
                          >
                          >
                          >
                          > _____
                          >
                          > To: nservicebus@ yahoogroups. com
                          > From: thesoftwaresimplist @...
                          > Date: Wed, 21 May 2008 01:27:43 +0300
                          > Subject: RE: [nservicebus] Handling failures
                          >
                          > The retry count doesn’t have to do with message delivery. MSMQ
                          handles that independently of nServiceBus.
                          >
                          >
                          >
                          > When a transaction is thrown during any part of processing the
                          message, then we retry the HANDLING of the message. Deserialization
                          exceptions are different â€" those are immediately transferred to the
                          error queue without any retries.
                          >
                          >
                          >
                          > The reason for this logic has to do with technological issues
                          preventing otherwise successful message handling â€" deadlocks in the
                          database are one example.
                          >
                          >
                          >
                          > Does that answer your question?
                          >
                          >
                          >
                          > --
                          >
                          > Udi Dahan - The Software Simplist
                          >
                          >
                          >
                          > From: nservicebus@ yahoogroups. com
                          [mailto:nservicebus@ yahoogroups. com] On Behalf Of pawel.pabich
                          > Sent: 21 May 2008 00:48
                          > To: nservicebus@ yahoogroups. com
                          > Subject: [nservicebus] Handling failures
                          >
                          >
                          >
                          > As far as I know if a message delivery fails then NServiceBus
                          retries
                          > immediately and keeps doing this until 5th attempt in a row fails.
                          If
                          > that happens then it sends the message to the error queue. Is there
                          > any setting that would instruct NServiceBus to wait some period of
                          > time between retries? Is there any other built-in way of handling
                          such
                          > cases when the first delivery fails?
                          >
                          > thanks
                          >
                          > Pawel
                          >
                          > No virus found in this incoming message.
                          > Checked by AVG.
                          > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                          20/05/2008 16:45
                          >
                          >
                          >
                          > _____
                          >
                          > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                          <http://video. se.msn.com/ v/sv-se/v. htm>
                          >
                          > No virus found in this incoming message.
                          > Checked by AVG.
                          > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                          20/05/2008 16:45
                          >
                          >
                          >
                          > No virus found in this incoming message.
                          > Checked by AVG.
                          > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                          20/05/2008 16:45
                          >
                          >
                          >
                          >
                          >
                          > No virus found in this incoming message.
                          > Checked by AVG.
                          > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                          20/05/2008 16:45
                          >


                        • pawel
                          By the external system I mean a system that the service needs to talk to synchronously when it processes a message. As an example, there is a system that uses
                          Message 12 of 20 , May 22, 2008
                          View Source
                          • 0 Attachment
                            By the external system I mean a system that the service needs to talk to synchronously when it processes a message.
                            As an example, there is a system that uses a home-grown database as the back-end  which leads to all sort of unexpected behaviours.

                            thanks

                            Pawel


                            From: "Udi Dahan" <thesoftwaresimplist@...>
                            Sent: 21 May 2008 14:52
                            To: nservicebus@yahoogroups.com
                            Subject: RE: [nservicebus] Handling failures


                            In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion?

                             

                            --

                            Udi Dahan - The Software Simplist

                             

                            From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                            Sent: 21 May 2008 16:14
                            To: nservicebus@ yahoogroups. com
                            Subject: RE: [nservicebus] Handling failures

                             

                            I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                            retry strategy might help a lot.

                            thanks again

                            Pawel


                            From: "Udi Dahan" <thesoftwaresimplist @...>
                            Sent: 21 May 2008 13:12
                            To: nservicebus@ yahoogroups. com
                            Subject: RE: [nservicebus] Handling failures

                            Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                             

                            When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                             

                            That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                             

                            While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                             

                            Hope that explains the reasoning a bit.

                             

                            --

                            Udi Dahan - The Software Simplist

                             

                            From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                            Sent: 21 May 2008 13:16
                            To: nservicebus@ yahoogroups. .com
                            Subject: RE: [nservicebus] Handling failures

                             

                            Hi,

                            If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                            1. The message is sent to the sender local queue.
                            2. The message is transported from sender local queue to recipient local queue.
                            3. The message is retrieved from the recipient local queue and the recipient processes the message.

                            The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                            5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                            support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                            that deals with failures when the message is actually being processed by the recipient.

                            thanks

                            Pawel


                            From: "Udi Dahan" <thesoftwaresimplist @...>
                            Sent: 21 May 2008 08:53
                            To: nservicebus@ yahoogroups. com
                            Subject: RE: [nservicebus] Handling failures

                            My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                             

                            That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                             

                            --

                            Udi Dahan - The Software Simplist

                             

                            From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of Andreas Öhlund
                            Sent: 21 May 2008 09:59
                            To: nservicebus@ yahoogroups. com
                            Subject: RE: [nservicebus] Handling failures

                             

                            Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                             
                            It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFail edEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                             
                            Thoughts?






                            To: nservicebus@ yahoogroups. com
                            From: thesoftwaresimplist @...
                            Date: Wed, 21 May 2008 01:27:43 +0300
                            Subject: RE: [nservicebus] Handling failures

                            The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                             

                            When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                             

                            The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                             

                            Does that answer your question?

                             

                            --

                            Udi Dahan - The Software Simplist

                             

                            From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel.pabich
                            Sent: 21 May 2008 00:48
                            To: nservicebus@ yahoogroups. com
                            Subject: [nservicebus] Handling failures

                             

                            As far as I know if a message delivery fails then NServiceBus retries
                            immediately and keeps doing this until 5th attempt in a row fails. If
                            that happens then it sends the message to the error queue. Is there
                            any setting that would instruct NServiceBus to wait some period of
                            time between retries? Is there any other built-in way of handling such
                            cases when the first delivery fails?

                            thanks

                            Pawel

                            No virus found in this incoming message.
                            Checked by AVG.
                            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                             


                            Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                            No virus found in this incoming message.
                            Checked by AVG.
                            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                             

                            No virus found in this incoming message.
                            Checked by AVG.
                            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                             

                            No virus found in this incoming message.
                            Checked by AVG.
                            Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45


                          • Udi Dahan
                            Now I understand what your scenario is! The way that external integration of that kind is handled is simple: Your application logic sits in one autonomous
                            Message 13 of 20 , May 22, 2008
                            View Source
                            • 0 Attachment

                              Now I understand what your scenario is!

                               

                              The way that external integration of that kind is handled is simple:

                               

                              Your application logic sits in one autonomous component. Another autonomous component serves as an anti-corruption layer, actually communicating with the external system. Your logic sends a message to the other AC. That AC handles the message and calls the synchronous service. Should there be a problem there, an exception will cause a retry. You can define the number of retries for that AC separately from everything else.

                               

                              If you want to inject a time component to the process, use sagas and the timeout manager.

                               

                              Does that help?

                               

                              --

                              Udi Dahan - The Software Simplist

                               

                              From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                              Sent: 22 May 2008 13:23
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                               

                              By the external system I mean a system that the service needs to talk to synchronously when it processes a message.
                              As an example, there is a system that uses a home-grown database as the back-end  which leads to all sort of unexpected behaviours.

                              thanks

                              Pawel


                              From: "Udi Dahan" <thesoftwaresimplist@...>
                              Sent: 21 May 2008 14:52
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                              In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion?

                               

                              --

                              Udi Dahan - The Software Simplist

                               

                              From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                              Sent: 21 May 2008 16:14
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                               

                              I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                              retry strategy might help a lot.

                              thanks again

                              Pawel


                              From: "Udi Dahan" <thesoftwaresimplist@...>
                              Sent: 21 May 2008 13:12
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                              Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                               

                              When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                               

                              That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                               

                              While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                               

                              Hope that explains the reasoning a bit.

                               

                              --

                              Udi Dahan - The Software Simplist

                               

                              From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                              Sent: 21 May 2008 13:16
                              To: nservicebus@yahoogroups..com
                              Subject: RE: [nservicebus] Handling failures

                               

                              Hi,

                              If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                              1. The message is sent to the sender local queue.
                              2. The message is transported from sender local queue to recipient local queue.
                              3. The message is retrieved from the recipient local queue and the recipient processes the message.

                              The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                              5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                              support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                              that deals with failures when the message is actually being processed by the recipient.

                              thanks

                              Pawel


                              From: "Udi Dahan" <thesoftwaresimplist@...>
                              Sent: 21 May 2008 08:53
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                              My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                               

                              That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                               

                              --

                              Udi Dahan - The Software Simplist

                               

                              From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                              Sent: 21 May 2008 09:59
                              To: nservicebus@yahoogroups.com
                              Subject: RE: [nservicebus] Handling failures

                               

                              Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                               
                              It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                               
                              Thoughts?







                              To: nservicebus@yahoogroups.com
                              From: thesoftwaresimplist@...
                              Date: Wed, 21 May 2008 01:27:43 +0300
                              Subject: RE: [nservicebus] Handling failures

                              The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                               

                              When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                               

                              The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                               

                              Does that answer your question?

                               

                              --

                              Udi Dahan - The Software Simplist

                               

                              From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                              Sent: 21 May 2008 00:48
                              To: nservicebus@yahoogroups.com
                              Subject: [nservicebus] Handling failures

                               

                              As far as I know if a message delivery fails then NServiceBus retries
                              immediately and keeps doing this until 5th attempt in a row fails. If
                              that happens then it sends the message to the error queue. Is there
                              any setting that would instruct NServiceBus to wait some period of
                              time between retries? Is there any other built-in way of handling such
                              cases when the first delivery fails?

                              thanks

                              Pawel

                              No virus found in this incoming message.
                              Checked by AVG.
                              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                               


                              Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                              No virus found in this incoming message.
                              Checked by AVG.
                              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                               

                              No virus found in this incoming message.
                              Checked by AVG.
                              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                               

                              No virus found in this incoming message.
                              Checked by AVG.
                              Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                               

                              No virus found in this incoming message.
                              Checked by AVG.
                              Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date: 21/05/2008 17:34

                            • Eric Hauser
                              I d rather not have to define an IReplayableMessage interface to do something like that, but you have to store the replay information somewhere. You could do
                              Message 14 of 20 , May 22, 2008
                              View Source
                              • 0 Attachment
                                I'd rather not have to define an IReplayableMessage interface to do
                                something like that, but you have to store the replay information
                                somewhere. You could do it entirely in memory and not send the
                                messages back to the bus so you don't have to do that.

                                Currently, there is no way to add/lookup message headers for a
                                message right? I believe the MSMQ transport uses a message envelope,
                                but I don't think headers are something that it exposes. It would be
                                nice to have this functionality.


                                --- In nservicebus@yahoogroups.com, "Udi Dahan"
                                <thesoftwaresimplist@...> wrote:
                                >
                                > That’s an interesting thought â€" using nServiceBus under the
                                ITransport level to implement that functionality. It would break
                                configuration based on type, though.
                                >
                                >
                                >
                                > --
                                >
                                > Udi Dahan - The Software Simplist
                                >
                                >
                                >
                                > From: nservicebus@yahoogroups.com
                                [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                > Sent: 22 May 2008 03:52
                                > To: nservicebus@yahoogroups.com
                                > Subject: [nservicebus] Re: Handling failures
                                >
                                >
                                >
                                > I don't understand why retry handling would need to be part of the
                                > framework? If you wanted to do that, couldn't you just do something
                                > like (prototype code):
                                >
                                > public interface IReplayableMessage : IMessage
                                > {
                                > int RetryCount { get; set; }
                                > DateTime LastReplayAttempt { get; set; }
                                > }
                                >
                                > public abstract class RetryMessageHandler<TMessage> :
                                > IMessageHandler<TMessage>
                                > where TMessage : IReplayableMessage
                                > {
                                > public IBus Bus { get; set; }
                                >
                                > public void HandleMessage(TMessage message)
                                > {
                                > if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                                > {
                                > Bus.HandleMessageLater();
                                > return;
                                > }
                                > try
                                > {
                                > HandleMessageInternal(message);
                                > }
                                > catch (Exception e)
                                > {
                                > if (message.RetryCount < 5)
                                > {
                                > message.RetryCount = message.RetryCount++;
                                > message.LastReplayAttempt = DateTime.Now;
                                > Bus.Send(message);
                                > }
                                > else
                                > {
                                > .. log error ..
                                > }
                                > }
                                > }
                                >
                                > public abstract void HandleMessageInternal(TMessage message)
                                > }
                                >
                                > --- In nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com> , "Udi Dahan"
                                > <thesoftwaresimplist@> wrote:
                                > >
                                > > In the example I was talking about, what do you view as the
                                > external system? Or does the fact that I assume MSMQ communication
                                > lead you to that conclusion?
                                > >
                                > >
                                > >
                                > > --
                                > >
                                > > Udi Dahan - The Software Simplist
                                > >
                                > >
                                > >
                                > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com> ] On Behalf Of pawel
                                > > Sent: 21 May 2008 16:14
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > >
                                > >
                                > > I see your point and it makes sense as long as you control
                                external
                                > systems. If you don't and they are a bit flaky than a good
                                > > retry strategy might help a lot.
                                > >
                                > > thanks again
                                > >
                                > > Pawel
                                > >
                                > > _____
                                > >
                                > > From: "Udi Dahan" <thesoftwaresimplist@>
                                > > Sent: 21 May 2008 13:12
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > > Actually, when you send a message, it first goes to the outgoing
                                > queue on the sending machine. Once that successfully occurs, the
                                > sending thread is released to go about its business. Then MSMQ
                                tries
                                > to get the message to the specified queue on the target machine.
                                Once
                                > that occurs, the message is deleted from the outgoing queue of the
                                > sending machine. From that point on, the message is sitting in a
                                > queue waiting to be processed. When the target process retrieves
                                the
                                > message from that queue (its input queue), it processes it.
                                > >
                                > >
                                > >
                                > > When the DB detects a deadlock between two threads, it kicks one
                                > out, and lets the other proceed. Given that transactions are
                                limited
                                > in scope to a single message, it is rare that the same thread will
                                > hit a deadlock on the same data when it tries N micro/milliseconds
                                > later, and, even if so, half that chance that it will be selected
                                as
                                > the victim again. The N micros/millis is the time it takes to
                                > rollback the transaction and try again.
                                > >
                                > >
                                > >
                                > > That being said, you should be setting the appropriate number of
                                > threads in your autonomous components so that you get the maximal
                                > throughput while maintaining minimal contention. The more threads
                                are
                                > NOT the better. Also, poor DB design can increase the chance of
                                > contention, as can processing large messages where transactions are
                                > open longer. This is all applicative design, and has little to do
                                > with a retry strategy.
                                > >
                                > >
                                > >
                                > > While I can’t categorically say that a retry strategy isnâ
                                €™t
                                > needed, I can say that I’ve handled very, VERY heavy loads
                                without
                                > needing one, just by profiling, setting the appropriate number of
                                > threads, and â€Å"just” by decreasing message size (as my
                                InfoQ
                                > article explained).
                                > >
                                > >
                                > >
                                > > Hope that explains the reasoning a bit.
                                > >
                                > >
                                > >
                                > > --
                                > >
                                > > Udi Dahan - The Software Simplist
                                > >
                                > >
                                > >
                                > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com> ] On Behalf Of pawel
                                > > Sent: 21 May 2008 13:16
                                > > To: nservicebus@
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > >
                                > >
                                > > Hi,
                                > >
                                > > If I understand correctly then whenever a message is sent using
                                > MSMQ we deal with 3 distinct steps.
                                > >
                                > > 1. The message is sent to the sender local queue.
                                > > 2. The message is transported from sender local queue to
                                recipient
                                > local queue.
                                > > 3. The message is retrieved from the recipient local queue and
                                the
                                > recipient processes the message.
                                > >
                                > > The re-try strategy that I'm talking about is related to the 3rd
                                > step. If a database is under heavy load then re-trying
                                > > 5 times in a row (without waiting between attempts) is not going
                                to
                                > help and MSMQ itself does not provide any
                                > > support at this stage(except transactional processing). I was
                                > wondering if there is a more sophisticated built-in strategy
                                > > that deals with failures when the message is actually being
                                > processed by the recipient.
                                > >
                                > > thanks
                                > >
                                > > Pawel
                                > >
                                > > _____
                                > >
                                > > From: "Udi Dahan" <thesoftwaresimplist@>
                                > > Sent: 21 May 2008 08:53
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > > My answer was to say that nServiceBus currently doesn’t do
                                those
                                > kinds of retry strategies â€" MSMQ does that by itself.
                                > >
                                > >
                                > >
                                > > That’s one example as to a case where a different
                                transport
                                > implementation would expose a different extensibility model.
                                > >
                                > >
                                > >
                                > > --
                                > >
                                > > Udi Dahan - The Software Simplist
                                > >
                                > >
                                > >
                                > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com> ] On Behalf Of Andreas Ãâ€"hlund
                                > > Sent: 21 May 2008 09:59
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > >
                                > >
                                > > Isn't Pawel asking about some kind of "Retry strategy". The
                                current
                                > one is "Retry 5 time with no wait". I've seen solutions that sleeps
                                a
                                > while before retrying ( and perhaps extends the "sleep period"
                                based
                                > on number of failed retries)
                                > >
                                > > It would also be nice if the bus or transport could expose
                                > an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know
                                > you could read the error queue but this would be a more transport
                                > agnostic solution?)
                                > >
                                > > Thoughts?
                                > >
                                > >
                                > >
                                > >
                                > >
                                > >
                                > >
                                > > _____
                                > >
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > From: thesoftwaresimplist@
                                > > Date: Wed, 21 May 2008 01:27:43 +0300
                                > > Subject: RE: [nservicebus] Handling failures
                                > >
                                > > The retry count doesn’t have to do with message delivery.
                                MSMQ
                                > handles that independently of nServiceBus.
                                > >
                                > >
                                > >
                                > > When a transaction is thrown during any part of processing the
                                > message, then we retry the HANDLING of the message. Deserialization
                                > exceptions are different â€" those are immediately transferred
                                to the
                                > error queue without any retries.
                                > >
                                > >
                                > >
                                > > The reason for this logic has to do with technological issues
                                > preventing otherwise successful message handling â€" deadlocks
                                in the
                                > database are one example.
                                > >
                                > >
                                > >
                                > > Does that answer your question?
                                > >
                                > >
                                > >
                                > > --
                                > >
                                > > Udi Dahan - The Software Simplist
                                > >
                                > >
                                > >
                                > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com> ] On Behalf Of pawel.pabich
                                > > Sent: 21 May 2008 00:48
                                > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                40yahoogroups.com>
                                > > Subject: [nservicebus] Handling failures
                                > >
                                > >
                                > >
                                > > As far as I know if a message delivery fails then NServiceBus
                                > retries
                                > > immediately and keeps doing this until 5th attempt in a row
                                fails.
                                > If
                                > > that happens then it sends the message to the error queue. Is
                                there
                                > > any setting that would instruct NServiceBus to wait some period of
                                > > time between retries? Is there any other built-in way of handling
                                > such
                                > > cases when the first delivery fails?
                                > >
                                > > thanks
                                > >
                                > > Pawel
                                > >
                                > > No virus found in this incoming message.
                                > > Checked by AVG.
                                > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                > 20/05/2008 16:45
                                > >
                                > >
                                > >
                                > > _____
                                > >
                                > > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                                > <http://video.se.msn.com/v/sv-se/v.htm>
                                > >
                                > > No virus found in this incoming message.
                                > > Checked by AVG.
                                > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                > 20/05/2008 16:45
                                > >
                                > >
                                > >
                                > > No virus found in this incoming message.
                                > > Checked by AVG.
                                > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                > 20/05/2008 16:45
                                > >
                                > >
                                > >
                                > >
                                > >
                                > > No virus found in this incoming message.
                                > > Checked by AVG.
                                > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                > 20/05/2008 16:45
                                > >
                                >
                                >
                                >
                                > No virus found in this incoming message.
                                > Checked by AVG.
                                > Version: 8.0.100 / Virus Database: 269.23.21/1458 - Release Date:
                                21/05/2008 07:21
                                >
                              • pawel
                                From my perspective this is like adding another level of indirection. At the end of the day there must be a piece of code that handles retries and it does not
                                Message 15 of 20 , May 22, 2008
                                View Source
                                • 0 Attachment
                                  From my perspective this is like adding another level of indirection.
                                  At the end of the day there must be a piece of code that handles retries and
                                  it does not really matter if it lives inside the message handler or in a separate component.
                                  By keeping the retry logic inside the message handler I can piggyback on the existing ambient transaction and MSMQ as
                                  the reliable storage.

                                  Pawel


                                  From: "Udi Dahan" <thesoftwaresimplist@...>
                                  Sent: 22 May 2008 13:25
                                  To: nservicebus@yahoogroups.com
                                  Subject: RE: [nservicebus] Handling failures


                                  Now I understand what your scenario is!

                                   

                                  The way that external integration of that kind is handled is simple:

                                   

                                  Your application logic sits in one autonomous component. Another autonomous component serves as an anti-corruption layer, actually communicating with the external system. Your logic sends a message to the other AC. That AC handles the message and calls the synchronous service. Should there be a problem there, an exception will cause a retry. You can define the number of retries for that AC separately from everything else.

                                   

                                  If you want to inject a time component to the process, use sagas and the timeout manager.

                                   

                                  Does that help?

                                   

                                  --

                                  Udi Dahan - The Software Simplist

                                   

                                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                                  Sent: 22 May 2008 13:23
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                   

                                  By the external system I mean a system that the service needs to talk to synchronously when it processes a message.
                                  As an example, there is a system that uses a home-grown database as the back-end  which leads to all sort of unexpected behaviours.

                                  thanks

                                  Pawel


                                  From: "Udi Dahan" <thesoftwaresimplist @...>
                                  Sent: 21 May 2008 14:52
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                  In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion?

                                   

                                  --

                                  Udi Dahan - The Software Simplist

                                   

                                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                                  Sent: 21 May 2008 16:14
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                   

                                  I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                                  retry strategy might help a lot.

                                  thanks again

                                  Pawel


                                  From: "Udi Dahan" <thesoftwaresimplist @...>
                                  Sent: 21 May 2008 13:12
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                  Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                                   

                                  When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                                   

                                  That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                                   

                                  While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                                   

                                  Hope that explains the reasoning a bit.

                                   

                                  --

                                  Udi Dahan - The Software Simplist

                                   

                                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel
                                  Sent: 21 May 2008 13:16
                                  To: nservicebus@ yahoogroups. .com
                                  Subject: RE: [nservicebus] Handling failures

                                   

                                  Hi,

                                  If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                                  1. The message is sent to the sender local queue.
                                  2. The message is transported from sender local queue to recipient local queue.
                                  3. The message is retrieved from the recipient local queue and the recipient processes the message.

                                  The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                                  5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                                  support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                                  that deals with failures when the message is actually being processed by the recipient.

                                  thanks

                                  Pawel


                                  From: "Udi Dahan" <thesoftwaresimplist @...>
                                  Sent: 21 May 2008 08:53
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                  My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                                   

                                  That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                                   

                                  --

                                  Udi Dahan - The Software Simplist

                                   

                                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of Andreas Öhlund
                                  Sent: 21 May 2008 09:59
                                  To: nservicebus@ yahoogroups. com
                                  Subject: RE: [nservicebus] Handling failures

                                   

                                  Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                                   
                                  It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFail edEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                                   
                                  Thoughts?







                                  To: nservicebus@ yahoogroups. com
                                  From: thesoftwaresimplist @...
                                  Date: Wed, 21 May 2008 01:27:43 +0300
                                  Subject: RE: [nservicebus] Handling failures

                                  The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                                   

                                  When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                                   

                                  The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                                   

                                  Does that answer your question?

                                   

                                  --

                                  Udi Dahan - The Software Simplist

                                   

                                  From: nservicebus@ yahoogroups. com [mailto:nservicebus @yahoogroups. com] On Behalf Of pawel.pabich
                                  Sent: 21 May 2008 00:48
                                  To: nservicebus@ yahoogroups. com
                                  Subject: [nservicebus] Handling failures

                                   

                                  As far as I know if a message delivery fails then NServiceBus retries
                                  immediately and keeps doing this until 5th attempt in a row fails. If
                                  that happens then it sends the message to the error queue. Is there
                                  any setting that would instruct NServiceBus to wait some period of
                                  time between retries? Is there any other built-in way of handling such
                                  cases when the first delivery fails?

                                  thanks

                                  Pawel

                                  No virus found in this incoming message.
                                  Checked by AVG.
                                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                   


                                  Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                                  No virus found in this incoming message.
                                  Checked by AVG.
                                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                   

                                  No virus found in this incoming message.
                                  Checked by AVG.
                                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                   

                                  No virus found in this incoming message.
                                  Checked by AVG.
                                  Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                   

                                  No virus found in this incoming message.
                                  Checked by AVG.
                                  Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date: 21/05/2008 17:34


                                • Nathan Stults
                                  I’d vote for exposing custom headers as well, somewhere to put and retrieve info that is needed by the messaging system and not by the domain. From:
                                  Message 16 of 20 , May 22, 2008
                                  View Source
                                  • 0 Attachment

                                    I’d vote for exposing custom headers as well, somewhere to put and retrieve info that is needed by the messaging system and not by the domain.

                                     

                                    From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                    Sent: Thursday, May 22, 2008 7:14 AM
                                    To: nservicebus@yahoogroups.com
                                    Subject: [nservicebus] Re: Handling failures

                                     

                                    I'd rather not have to define an IReplayableMessage interface to do
                                    something like that, but you have to store the replay information
                                    somewhere. You could do it entirely in memory and not send the
                                    messages back to the bus so you don't have to do that.

                                    Currently, there is no way to add/lookup message headers for a
                                    message right? I believe the MSMQ transport uses a message envelope,
                                    but I don't think headers are something that it exposes. It would be
                                    nice to have this functionality.

                                    --- In nservicebus@yahoogroups.com, "Udi Dahan"
                                    <thesoftwaresimplist@...> wrote:

                                    >
                                    > That’s an interesting thought â€" using nServiceBus under the
                                    ITransport level to implement that functionality. It would break
                                    configuration based on type, though.
                                    >
                                    >
                                    >
                                    > --
                                    >
                                    > Udi Dahan - The Software Simplist
                                    >
                                    >
                                    >
                                    > From: nservicebus@yahoogroups.com
                                    [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                    > Sent: 22 May 2008 03:52
                                    > To: nservicebus@yahoogroups.com
                                    > Subject: [nservicebus] Re: Handling failures
                                    >
                                    >
                                    >
                                    > I don't understand why retry handling would need to be part of the
                                    > framework? If you wanted to do that, couldn't you just do something
                                    > like (prototype code):
                                    >
                                    > public interface IReplayableMessage : IMessage
                                    > {
                                    > int RetryCount { get; set; }
                                    > DateTime LastReplayAttempt { get; set; }
                                    > }
                                    >
                                    > public abstract class RetryMessageHandler<TMessage> :
                                    > IMessageHandler<TMessage>
                                    > where TMessage : IReplayableMessage
                                    > {
                                    > public IBus Bus { get; set; }
                                    >
                                    > public void HandleMessage(TMessage message)
                                    > {
                                    > if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                                    > {
                                    > Bus.HandleMessageLater();
                                    > return;
                                    > }
                                    > try
                                    > {
                                    > HandleMessageInternal(message);
                                    > }
                                    > catch (Exception e)
                                    > {
                                    > if (message.RetryCount < 5)
                                    > {
                                    > message.RetryCount = message.RetryCount++;
                                    > message.LastReplayAttempt = DateTime.Now;
                                    > Bus.Send(message);
                                    > }
                                    > else
                                    > {
                                    > .. log error ..
                                    > }
                                    > }
                                    > }
                                    >
                                    > public abstract void HandleMessageInternal(TMessage message)
                                    > }
                                    >
                                    > --- In nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com> , "Udi Dahan"
                                    > <thesoftwaresimplist@> wrote:
                                    > >
                                    > > In the example I was talking about, what do you view as the
                                    > external system? Or does the fact that I assume MSMQ communication
                                    > lead you to that conclusion?
                                    > >
                                    > >
                                    > >
                                    > > --
                                    > >
                                    > > Udi Dahan - The Software Simplist
                                    > >
                                    > >
                                    > >
                                    > > From: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > [mailto:nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com> ] On Behalf Of pawel
                                    > > Sent: 21 May 2008 16:14
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > >
                                    > >
                                    > > I see your point and it makes sense as long as you control
                                    external
                                    > systems. If you don't and they are a bit flaky than a good
                                    > > retry strategy might help a lot.
                                    > >
                                    > > thanks again
                                    > >
                                    > > Pawel
                                    > >
                                    > > _____
                                    > >
                                    > > From: "Udi Dahan" <thesoftwaresimplist@>
                                    > > Sent: 21 May 2008 13:12
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > > Actually, when you send a message, it first goes to the outgoing
                                    > queue on the sending machine. Once that successfully occurs, the
                                    > sending thread is released to go about its business. Then MSMQ
                                    tries
                                    > to get the message to the specified queue on the target machine.
                                    Once
                                    > that occurs, the message is deleted from the outgoing queue of the
                                    > sending machine. From that point on, the message is sitting in a
                                    > queue waiting to be processed. When the target process retrieves
                                    the
                                    > message from that queue (its input queue), it processes it.
                                    > >
                                    > >
                                    > >
                                    > > When the DB detects a deadlock between two threads, it kicks one
                                    > out, and lets the other proceed. Given that transactions are
                                    limited
                                    > in scope to a single message, it is rare that the same thread will
                                    > hit a deadlock on the same data when it tries N micro/milliseconds
                                    > later, and, even if so, half that chance that it will be selected
                                    as
                                    > the victim again. The N micros/millis is the time it takes to
                                    > rollback the transaction and try again.
                                    > >
                                    > >
                                    > >
                                    > > That being said, you should be setting the appropriate number of
                                    > threads in your autonomous components so that you get the maximal
                                    > throughput while maintaining minimal contention. The more threads
                                    are
                                    > NOT the better. Also, poor DB design can increase the chance of
                                    > contention, as can processing large messages where transactions are
                                    > open longer. This is all applicative design, and has little to do
                                    > with a retry strategy.
                                    > >
                                    > >
                                    > >
                                    > > While I can’t categorically say that a retry strategy isnâ
                                    €™t
                                    > needed, I can say that I’ve handled very, VERY heavy loads
                                    without
                                    > needing one, just by profiling, setting the appropriate number of
                                    > threads, and â€Å"just” by decreasing message size (as my
                                    InfoQ
                                    > article explained).
                                    > >
                                    > >
                                    > >
                                    > > Hope that explains the reasoning a bit.
                                    > >
                                    > >
                                    > >
                                    > > --
                                    > >
                                    > > Udi Dahan - The Software Simplist
                                    > >
                                    > >
                                    > >
                                    > > From: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > [mailto:nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com> ] On Behalf Of pawel
                                    > > Sent: 21 May 2008 13:16
                                    > > To: nservicebus@
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > >
                                    > >
                                    > > Hi,
                                    > >
                                    > > If I understand correctly then whenever a message is sent using
                                    > MSMQ we deal with 3 distinct steps.
                                    > >
                                    > > 1. The message is sent to the sender local queue.
                                    > > 2. The message is transported from sender local queue to
                                    recipient
                                    > local queue.
                                    > > 3. The message is retrieved from the recipient local queue and
                                    the
                                    > recipient processes the message.
                                    > >
                                    > > The re-try strategy that I'm talking about is related to the 3rd
                                    > step. If a database is under heavy load then re-trying
                                    > > 5 times in a row (without waiting between attempts) is not going
                                    to
                                    > help and MSMQ itself does not provide any
                                    > > support at this stage(except transactional processing). I was
                                    > wondering if there is a more sophisticated built-in strategy
                                    > > that deals with failures when the message is actually being
                                    > processed by the recipient.
                                    > >
                                    > > thanks
                                    > >
                                    > > Pawel
                                    > >
                                    > > _____
                                    > >
                                    > > From: "Udi Dahan" <thesoftwaresimplist@>
                                    > > Sent: 21 May 2008 08:53
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > > My answer was to say that nServiceBus currently doesn’t do
                                    those
                                    > kinds of retry strategies â€" MSMQ does that by itself.
                                    > >
                                    > >
                                    > >
                                    > > That’s one example as to a case where a different
                                    transport
                                    > implementation would expose a different extensibility model.
                                    > >
                                    > >
                                    > >
                                    > > --
                                    > >
                                    > > Udi Dahan - The Software Simplist
                                    > >
                                    > >
                                    > >
                                    > > From: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > [mailto:nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com> ] On Behalf Of Andreas Ãâ€"hlund
                                    > > Sent: 21 May 2008 09:59
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > >
                                    > >
                                    > > Isn't Pawel asking about some kind of "Retry strategy". The
                                    current
                                    > one is "Retry 5 time with no wait". I've seen solutions that
                                    sleeps
                                    a
                                    > while before retrying ( and perhaps extends the "sleep period"
                                    based
                                    > on number of failed retries)
                                    > >
                                    > > It would also be nice if the bus or transport could expose
                                    > an "MessageRetriedEvent" and a "MessageHandlingFailedEvent"
                                    (I know
                                    > you could read the error queue but this would be a more transport
                                    > agnostic solution?)
                                    > >
                                    > > Thoughts?
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > > _____
                                    > >
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > From: thesoftwaresimplist@
                                    > > Date: Wed, 21 May 2008 01:27:43 +0300
                                    > > Subject: RE: [nservicebus] Handling failures
                                    > >
                                    > > The retry count doesn’t have to do with message delivery.
                                    MSMQ
                                    > handles that independently of nServiceBus.
                                    > >
                                    > >
                                    > >
                                    > > When a transaction is thrown during any part of processing the
                                    > message, then we retry the HANDLING of the message. Deserialization
                                    > exceptions are different â€" those are immediately transferred
                                    to the
                                    > error queue without any retries.
                                    > >
                                    > >
                                    > >
                                    > > The reason for this logic has to do with technological issues
                                    > preventing otherwise successful message handling â€" deadlocks
                                    in the
                                    > database are one example.
                                    > >
                                    > >
                                    > >
                                    > > Does that answer your question?
                                    > >
                                    > >
                                    > >
                                    > > --
                                    > >
                                    > > Udi Dahan - The Software Simplist
                                    > >
                                    > >
                                    > >
                                    > > From: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > [mailto:nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com> ] On Behalf Of pawel.pabich
                                    > > Sent: 21 May 2008 00:48
                                    > > To: nservicebus@yahoogroups.com
                                    <mailto:nservicebus%
                                    40yahoogroups.com>
                                    > > Subject: [nservicebus] Handling failures
                                    > >
                                    > >
                                    > >
                                    > > As far as I know if a message delivery fails then NServiceBus
                                    > retries
                                    > > immediately and keeps doing this until 5th attempt in a row
                                    fails.
                                    > If
                                    > > that happens then it sends the message to the error queue. Is
                                    there
                                    > > any setting that would instruct NServiceBus to wait some period of
                                    > > time between retries? Is there any other built-in way of handling
                                    > such
                                    > > cases when the first delivery fails?
                                    > >
                                    > > thanks
                                    > >
                                    > > Pawel
                                    > >
                                    > > No virus found in this incoming message.
                                    > > Checked by AVG.
                                    > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                    > 20/05/2008 16:45
                                    > >
                                    > >
                                    > >
                                    > > _____
                                    > >
                                    > > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                                    > <http://video.se.msn.com/v/sv-se/v.htm>
                                    > >
                                    > > No virus found in this incoming message.
                                    > > Checked by AVG.
                                    > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                    > 20/05/2008 16:45
                                    > >
                                    > >
                                    > >
                                    > > No virus found in this incoming message.
                                    > > Checked by AVG.
                                    > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                    > 20/05/2008 16:45
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > > No virus found in this incoming message.
                                    > > Checked by AVG.
                                    > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                    > 20/05/2008 16:45
                                    > >
                                    >
                                    >
                                    >
                                    > No virus found in this incoming message.
                                    > Checked by AVG.
                                    > Version: 8.0.100 / Virus Database: 269.23.21/1458 - Release Date:
                                    21/05/2008 07:21
                                    >

                                  • Udi Dahan
                                    Currently there’s TransportMessage which sits on the boundary between the bus and the transport, exposing to the bus in a technology agnostic way the
                                    Message 17 of 20 , May 22, 2008
                                    View Source
                                    • 0 Attachment

                                      Currently there’s TransportMessage which sits on the boundary between the bus and the transport, exposing to the bus in a technology agnostic way the properties it cares about. From what I’m hearing, you’re asking to customize the behaviour of the transport with respect to the underlying technology, but not so much from the bus’ perspective. That sounds like you should just write your own custom transport.

                                       

                                      If any of the above assumptions are incorrect, please let me know.

                                       

                                      Thanks,

                                       

                                      --

                                      Udi Dahan - The Software Simplist

                                       

                                      From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Nathan Stults
                                      Sent: 22 May 2008 18:30
                                      To: nservicebus@yahoogroups.com
                                      Subject: RE: [nservicebus] Re: Handling failures

                                       

                                      I’d vote for exposing custom headers as well, somewhere to put and retrieve info that is needed by the messaging system and not by the domain.

                                       

                                      From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                      Sent: Thursday, May 22, 2008 7:14 AM
                                      To: nservicebus@yahoogroups.com
                                      Subject: [nservicebus] Re: Handling failures

                                       

                                      I'd rather not have to define an IReplayableMessage interface to do
                                      something like that, but you have to store the replay information
                                      somewhere. You could do it entirely in memory and not send the
                                      messages back to the bus so you don't have to do that.

                                      Currently, there is no way to add/lookup message headers for a
                                      message right? I believe the MSMQ transport uses a message envelope,
                                      but I don't think headers are something that it exposes. It would be
                                      nice to have this functionality.

                                      --- In nservicebus@yahoogroups.com, "Udi Dahan"
                                      <thesoftwaresimplist@...> wrote:

                                      >
                                      > That’s an interesting thought â€" using nServiceBus under the
                                      ITransport level to implement that functionality. It would break
                                      configuration based on type, though.
                                      >
                                      >
                                      >
                                      > --
                                      >
                                      > Udi Dahan - The Software Simplist
                                      >
                                      >
                                      >
                                      > From: nservicebus@yahoogroups.com
                                      [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                      > Sent: 22 May 2008 03:52
                                      > To: nservicebus@yahoogroups.com
                                      > Subject: [nservicebus] Re: Handling failures
                                      >
                                      >
                                      >
                                      > I don't understand why retry handling would need to be part of the
                                      > framework? If you wanted to do that, couldn't you just do something
                                      > like (prototype code):
                                      >
                                      > public interface IReplayableMessage : IMessage
                                      > {
                                      > int RetryCount { get; set; }
                                      > DateTime LastReplayAttempt { get; set; }
                                      > }
                                      >
                                      > public abstract class RetryMessageHandler<TMessage> :
                                      > IMessageHandler<TMessage>
                                      > where TMessage : IReplayableMessage
                                      > {
                                      > public IBus Bus { get; set; }
                                      >
                                      > public void HandleMessage(TMessage message)
                                      > {
                                      > if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                                      > {
                                      > Bus.HandleMessageLater();
                                      > return;
                                      > }
                                      > try
                                      > {
                                      > HandleMessageInternal(message);
                                      > }
                                      > catch (Exception e)
                                      > {
                                      > if (message.RetryCount < 5)
                                      > {
                                      > message.RetryCount = message.RetryCount++;
                                      > message.LastReplayAttempt = DateTime.Now;
                                      > Bus.Send(message);
                                      > }
                                      > else
                                      > {
                                      > .. log error ..
                                      > }
                                      > }
                                      > }
                                      >
                                      > public abstract void HandleMessageInternal(TMessage message)
                                      > }
                                      >
                                      > --- In nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com> , "Udi Dahan"
                                      > <thesoftwaresimplist@> wrote:
                                      > >
                                      > > In the example I was talking about, what do you view as the
                                      > external system? Or does the fact that I assume MSMQ communication
                                      > lead you to that conclusion?
                                      > >
                                      > >
                                      > >
                                      > > --
                                      > >
                                      > > Udi Dahan - The Software Simplist
                                      > >
                                      > >
                                      > >
                                      > > From: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > [mailto:nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com> ] On Behalf Of pawel
                                      > > Sent: 21 May 2008 16:14
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > >
                                      > >
                                      > > I see your point and it makes sense as long as you control
                                      external
                                      > systems. If you don't and they are a bit flaky than a good
                                      > > retry strategy might help a lot.
                                      > >
                                      > > thanks again
                                      > >
                                      > > Pawel
                                      > >
                                      > > _____
                                      > >
                                      > > From: "Udi Dahan" <thesoftwaresimplist@>
                                      > > Sent: 21 May 2008 13:12
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > > Actually, when you send a message, it first goes to the outgoing
                                      > queue on the sending machine. Once that successfully occurs, the
                                      > sending thread is released to go about its business. Then MSMQ
                                      tries
                                      > to get the message to the specified queue on the target machine.
                                      Once
                                      > that occurs, the message is deleted from the outgoing queue of the
                                      > sending machine. From that point on, the message is sitting in a
                                      > queue waiting to be processed. When the target process retrieves
                                      the
                                      > message from that queue (its input queue), it processes it.
                                      > >
                                      > >
                                      > >
                                      > > When the DB detects a deadlock between two threads, it kicks one
                                      > out, and lets the other proceed. Given that transactions are
                                      limited
                                      > in scope to a single message, it is rare that the same thread will
                                      > hit a deadlock on the same data when it tries N micro/milliseconds
                                      > later, and, even if so, half that chance that it will be selected
                                      as
                                      > the victim again. The N micros/millis is the time it takes to
                                      > rollback the transaction and try again.
                                      > >
                                      > >
                                      > >
                                      > > That being said, you should be setting the appropriate number of
                                      > threads in your autonomous components so that you get the maximal
                                      > throughput while maintaining minimal contention. The more threads
                                      are
                                      > NOT the better. Also, poor DB design can increase the chance of
                                      > contention, as can processing large messages where transactions are
                                      > open longer. This is all applicative design, and has little to do
                                      > with a retry strategy.
                                      > >
                                      > >
                                      > >
                                      > > While I can’t categorically say that a retry strategy isnâ
                                      €™t
                                      > needed, I can say that I’ve handled very, VERY heavy loads
                                      without
                                      > needing one, just by profiling, setting the appropriate number of
                                      > threads, and â€Å"just” by decreasing message size (as my
                                      InfoQ
                                      > article explained).
                                      > >
                                      > >
                                      > >
                                      > > Hope that explains the reasoning a bit.
                                      > >
                                      > >
                                      > >
                                      > > --
                                      > >
                                      > > Udi Dahan - The Software Simplist
                                      > >
                                      > >
                                      > >
                                      > > From: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > [mailto:nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com> ] On Behalf Of pawel
                                      > > Sent: 21 May 2008 13:16
                                      > > To: nservicebus@
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > >
                                      > >
                                      > > Hi,
                                      > >
                                      > > If I understand correctly then whenever a message is sent using
                                      > MSMQ we deal with 3 distinct steps.
                                      > >
                                      > > 1. The message is sent to the sender local queue.
                                      > > 2. The message is transported from sender local queue to
                                      recipient
                                      > local queue.
                                      > > 3. The message is retrieved from the recipient local queue and
                                      the
                                      > recipient processes the message.
                                      > >
                                      > > The re-try strategy that I'm talking about is related to the 3rd
                                      > step. If a database is under heavy load then re-trying
                                      > > 5 times in a row (without waiting between attempts) is not going
                                      to
                                      > help and MSMQ itself does not provide any
                                      > > support at this stage(except transactional processing). I was
                                      > wondering if there is a more sophisticated built-in strategy
                                      > > that deals with failures when the message is actually being
                                      > processed by the recipient.
                                      > >
                                      > > thanks
                                      > >
                                      > > Pawel
                                      > >
                                      > > _____
                                      > >
                                      > > From: "Udi Dahan" <thesoftwaresimplist@>
                                      > > Sent: 21 May 2008 08:53
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > > My answer was to say that nServiceBus currently doesn’t do
                                      those
                                      > kinds of retry strategies â€" MSMQ does that by itself.
                                      > >
                                      > >
                                      > >
                                      > > That’s one example as to a case where a different
                                      transport
                                      > implementation would expose a different extensibility model.
                                      > >
                                      > >
                                      > >
                                      > > --
                                      > >
                                      > > Udi Dahan - The Software Simplist
                                      > >
                                      > >
                                      > >
                                      > > From: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > [mailto:nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com> ] On Behalf Of Andreas Ãâ€"hlund
                                      > > Sent: 21 May 2008 09:59
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > >
                                      > >
                                      > > Isn't Pawel asking about some kind of "Retry strategy". The
                                      current
                                      > one is "Retry 5 time with no wait". I've seen solutions that
                                      sleeps
                                      a
                                      > while before retrying ( and perhaps extends the "sleep period"
                                      based
                                      > on number of failed retries)
                                      > >
                                      > > It would also be nice if the bus or transport could expose
                                      > an "MessageRetriedEvent" and a "MessageHandlingFailedEvent"
                                      (I know
                                      > you could read the error queue but this would be a more transport
                                      > agnostic solution?)
                                      > >
                                      > > Thoughts?
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > > _____
                                      > >
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > From: thesoftwaresimplist@
                                      > > Date: Wed, 21 May 2008 01:27:43 +0300
                                      > > Subject: RE: [nservicebus] Handling failures
                                      > >
                                      > > The retry count doesn’t have to do with message delivery.
                                      MSMQ
                                      > handles that independently of nServiceBus.
                                      > >
                                      > >
                                      > >
                                      > > When a transaction is thrown during any part of processing the
                                      > message, then we retry the HANDLING of the message. Deserialization
                                      > exceptions are different â€" those are immediately transferred
                                      to the
                                      > error queue without any retries.
                                      > >
                                      > >
                                      > >
                                      > > The reason for this logic has to do with technological issues
                                      > preventing otherwise successful message handling â€" deadlocks
                                      in the
                                      > database are one example.
                                      > >
                                      > >
                                      > >
                                      > > Does that answer your question?
                                      > >
                                      > >
                                      > >
                                      > > --
                                      > >
                                      > > Udi Dahan - The Software Simplist
                                      > >
                                      > >
                                      > >
                                      > > From: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > [mailto:nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com> ] On Behalf Of pawel.pabich
                                      > > Sent: 21 May 2008 00:48
                                      > > To: nservicebus@yahoogroups.com
                                      <mailto:nservicebus%
                                      40yahoogroups.com>
                                      > > Subject: [nservicebus] Handling failures
                                      > >
                                      > >
                                      > >
                                      > > As far as I know if a message delivery fails then NServiceBus
                                      > retries
                                      > > immediately and keeps doing this until 5th attempt in a row
                                      fails.
                                      > If
                                      > > that happens then it sends the message to the error queue. Is
                                      there
                                      > > any setting that would instruct NServiceBus to wait some period of
                                      > > time between retries? Is there any other built-in way of handling
                                      > such
                                      > > cases when the first delivery fails?
                                      > >
                                      > > thanks
                                      > >
                                      > > Pawel
                                      > >
                                      > > No virus found in this incoming message.
                                      > > Checked by AVG.
                                      > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                      > 20/05/2008 16:45
                                      > >
                                      > >
                                      > >
                                      > > _____
                                      > >
                                      > > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                                      > <http://video.se.msn.com/v/sv-se/v.htm>
                                      > >
                                      > > No virus found in this incoming message.
                                      > > Checked by AVG.
                                      > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                      > 20/05/2008 16:45
                                      > >
                                      > >
                                      > >
                                      > > No virus found in this incoming message.
                                      > > Checked by AVG.
                                      > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                      > 20/05/2008 16:45
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > > No virus found in this incoming message.
                                      > > Checked by AVG.
                                      > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                      > 20/05/2008 16:45
                                      > >
                                      >
                                      >
                                      >
                                      > No virus found in this incoming message.
                                      > Checked by AVG.
                                      > Version: 8.0.100 / Virus Database: 269.23.21/1458 - Release Date:
                                      21/05/2008 07:21
                                      >

                                      No virus found in this incoming message.
                                      Checked by AVG.
                                      Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date: 21/05/2008 17:34

                                    • Udi Dahan
                                      Just keep in mind what happens to that state (the number of retries) when your server restarts. Maybe the answer is “I don’t care, I’ll just start from
                                      Message 18 of 20 , May 22, 2008
                                      View Source
                                      • 0 Attachment

                                        Just keep in mind what happens to that state (the number of retries) when your server restarts. Maybe the answer is “I don’t care, I’ll just start from scratch”, but there are some protocols where you do need to know at which stage you failed, and in what way.

                                         

                                        One last thing, indirection helps move complexity around, a core capability in keeping your system testable and maintainable.

                                         

                                        Do you have a specific feature request in mind?

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                                        Sent: 22 May 2008 17:15
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                         

                                        From my perspective this is like adding another level of indirection.
                                        At the end of the day there must be a piece of code that handles retries and
                                        it does not really matter if it lives inside the message handler or in a separate component.
                                        By keeping the retry logic inside the message handler I can piggyback on the existing ambient transaction and MSMQ as
                                        the reliable storage.

                                        Pawel


                                        From: "Udi Dahan" <thesoftwaresimplist@...>
                                        Sent: 22 May 2008 13:25
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                        Now I understand what your scenario is!

                                         

                                        The way that external integration of that kind is handled is simple:

                                         

                                        Your application logic sits in one autonomous component. Another autonomous component serves as an anti-corruption layer, actually communicating with the external system. Your logic sends a message to the other AC. That AC handles the message and calls the synchronous service. Should there be a problem there, an exception will cause a retry. You can define the number of retries for that AC separately from everything else.

                                         

                                        If you want to inject a time component to the process, use sagas and the timeout manager.

                                         

                                        Does that help?

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                                        Sent: 22 May 2008 13:23
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                         

                                        By the external system I mean a system that the service needs to talk to synchronously when it processes a message.
                                        As an example, there is a system that uses a home-grown database as the back-end  which leads to all sort of unexpected behaviours.

                                        thanks

                                        Pawel


                                        From: "Udi Dahan" <thesoftwaresimplist@...>
                                        Sent: 21 May 2008 14:52
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                        In the example I was talking about, what do you view as the external system? Or does the fact that I assume MSMQ communication lead you to that conclusion?

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                                        Sent: 21 May 2008 16:14
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                         

                                        I see your point and it makes sense as long as you control external systems. If you don't and they are a bit flaky than a good
                                        retry strategy might help a lot.

                                        thanks again

                                        Pawel


                                        From: "Udi Dahan" <thesoftwaresimplist@...>
                                        Sent: 21 May 2008 13:12
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                        Actually, when you send a message, it first goes to the outgoing queue on the sending machine. Once that successfully occurs, the sending thread is released to go about its business. Then MSMQ tries to get the message to the specified queue on the target machine. Once that occurs, the message is deleted from the outgoing queue of the sending machine. From that point on, the message is sitting in a queue waiting to be processed. When the target process retrieves the message from that queue (its input queue), it processes it.

                                         

                                        When the DB detects a deadlock between two threads, it kicks one out, and lets the other proceed. Given that transactions are limited in scope to a single message, it is rare that the same thread will hit a deadlock on the same data when it tries N micro/milliseconds later, and, even if so, half that chance that it will be selected as the victim again. The N micros/millis is the time it takes to rollback the transaction and try again.

                                         

                                        That being said, you should be setting the appropriate number of threads in your autonomous components so that you get the maximal throughput while maintaining minimal contention. The more threads are NOT the better. Also, poor DB design can increase the chance of contention, as can processing large messages where transactions are open longer. This is all applicative design, and has little to do with a retry strategy.

                                         

                                        While I can’t categorically say that a retry strategy isn’t needed, I can say that I’ve handled very, VERY heavy loads without needing one, just by profiling, setting the appropriate number of threads, and “just” by decreasing message size (as my InfoQ article explained).

                                         

                                        Hope that explains the reasoning a bit.

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel
                                        Sent: 21 May 2008 13:16
                                        To: nservicebus@yahoogroups..com
                                        Subject: RE: [nservicebus] Handling failures

                                         

                                        Hi,

                                        If I understand correctly then whenever a message is sent using MSMQ we deal with 3 distinct steps.

                                        1. The message is sent to the sender local queue.
                                        2. The message is transported from sender local queue to recipient local queue.
                                        3. The message is retrieved from the recipient local queue and the recipient processes the message.

                                        The re-try strategy that I'm talking about is related to the 3rd step. If a database is under heavy load then re-trying
                                        5 times in a row (without waiting between attempts) is not going to help and MSMQ itself does not provide any
                                        support at this stage(except transactional processing). I was wondering if there is a more sophisticated built-in strategy
                                        that deals with failures when the message is actually being processed by the recipient.

                                        thanks

                                        Pawel


                                        From: "Udi Dahan" <thesoftwaresimplist@...>
                                        Sent: 21 May 2008 08:53
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                        My answer was to say that nServiceBus currently doesn’t do those kinds of retry strategies – MSMQ does that by itself.

                                         

                                        That’s one example as to a case where a different transport implementation would expose a different extensibility model.

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Andreas Öhlund
                                        Sent: 21 May 2008 09:59
                                        To: nservicebus@yahoogroups.com
                                        Subject: RE: [nservicebus] Handling failures

                                         

                                        Isn't Pawel asking about some kind of "Retry strategy". The current one is "Retry 5 time with no wait". I've seen solutions that sleeps a while before retrying ( and perhaps extends the "sleep period" based on number of failed retries)
                                         
                                        It would also be nice if the bus or transport could expose an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you could read the error queue but this would be a more transport agnostic solution?)
                                         
                                        Thoughts?








                                        To: nservicebus@yahoogroups.com
                                        From: thesoftwaresimplist@...
                                        Date: Wed, 21 May 2008 01:27:43 +0300
                                        Subject: RE: [nservicebus] Handling failures

                                        The retry count doesn’t have to do with message delivery. MSMQ handles that independently of nServiceBus.

                                         

                                        When a transaction is thrown during any part of processing the message, then we retry the HANDLING of the message. Deserialization exceptions are different – those are immediately transferred to the error queue without any retries.

                                         

                                        The reason for this logic has to do with technological issues preventing otherwise successful message handling – deadlocks in the database are one example.

                                         

                                        Does that answer your question?

                                         

                                        --

                                        Udi Dahan - The Software Simplist

                                         

                                        From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of pawel.pabich
                                        Sent: 21 May 2008 00:48
                                        To: nservicebus@yahoogroups.com
                                        Subject: [nservicebus] Handling failures

                                         

                                        As far as I know if a message delivery fails then NServiceBus retries
                                        immediately and keeps doing this until 5th attempt in a row fails. If
                                        that happens then it sends the message to the error queue. Is there
                                        any setting that would instruct NServiceBus to wait some period of
                                        time between retries? Is there any other built-in way of handling such
                                        cases when the first delivery fails?

                                        thanks

                                        Pawel

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                         


                                        Den perfekta mixen av nöjesnyheter livekonserter! MSN Video

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                         

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                         

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date: 20/05/2008 16:45

                                         

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date: 21/05/2008 17:34

                                         

                                        No virus found in this incoming message.
                                        Checked by AVG.
                                        Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date: 21/05/2008 17:34

                                      • Eric Hauser
                                        I m not referring to a transport specific header. Basically, I just want an IDictionary property called Headers on TransportMessage and a way
                                        Message 19 of 20 , May 22, 2008
                                        View Source
                                        • 0 Attachment
                                          I'm not referring to a transport specific header. Basically, I just
                                          want an IDictionary<string, string> property called Headers on
                                          TransportMessage and a way to access and mutate that property when
                                          processing an event handler. That way if I wanted to implement
                                          something like the dead letter channel I was discussing below, I
                                          would not have to make my messages implement a special interface.
                                          With this functionality, my sample could look like:


                                          public void HandleMessage(TMessage message)
                                          {
                                          var transportMessage = TransportMessage.For(message);
                                          int retryCount;
                                          transportMessage.Headers.TryGet("RetryCount", out retryCount);
                                          DateTime lastReplay;
                                          transportMessage.Headers.TryGet("LastReplayAttempt", out
                                          lastReplayAttempt);

                                          ... snip ...
                                          if (retryCount < 5)
                                          {
                                          transportMessage.Headers["RetryCount"] = retryCount++;
                                          transportMessage.Headers["LastReplayAttempt"] = DateTime.Now;
                                          Bus.Send(message);
                                          }
                                          }

                                          Service locator might not be the way to go for getting the transport
                                          message -- just throwing it out as an example. But this would be a
                                          transport independent way of storing some additional information
                                          about a message. If the individual transports wanted to put there
                                          headers in that dictionary (MSMQ headers, HTTP headers, SOAP headers,
                                          etc.), then another dictionary called TransportHeaders would probably
                                          be appropriate.


                                          --- In nservicebus@yahoogroups.com, "Udi Dahan"
                                          <thesoftwaresimplist@...> wrote:
                                          >
                                          > Currently there’s TransportMessage which sits on the boundary
                                          between the bus and the transport, exposing to the bus in a
                                          technology agnostic way the properties it cares about. From what
                                          I’m hearing, you’re asking to customize the behaviour of the
                                          transport with respect to the underlying technology, but not so much
                                          from the bus’ perspective. That sounds like you should just write
                                          your own custom transport.
                                          >
                                          >
                                          >
                                          > If any of the above assumptions are incorrect, please let me know.
                                          >
                                          >
                                          >
                                          > Thanks,
                                          >
                                          >
                                          >
                                          > --
                                          >
                                          > Udi Dahan - The Software Simplist
                                          >
                                          >
                                          >
                                          > From: nservicebus@yahoogroups.com
                                          [mailto:nservicebus@yahoogroups.com] On Behalf Of Nathan Stults
                                          > Sent: 22 May 2008 18:30
                                          > To: nservicebus@yahoogroups.com
                                          > Subject: RE: [nservicebus] Re: Handling failures
                                          >
                                          >
                                          >
                                          > I’d vote for exposing custom headers as well, somewhere to put
                                          and retrieve info that is needed by the messaging system and not by
                                          the domain.
                                          >
                                          >
                                          >
                                          > From: nservicebus@yahoogroups.com
                                          [mailto:nservicebus@yahoogroups.com] On Behalf Of Eric Hauser
                                          > Sent: Thursday, May 22, 2008 7:14 AM
                                          > To: nservicebus@yahoogroups.com
                                          > Subject: [nservicebus] Re: Handling failures
                                          >
                                          >
                                          >
                                          > I'd rather not have to define an IReplayableMessage interface to do
                                          > something like that, but you have to store the replay information
                                          > somewhere. You could do it entirely in memory and not send the
                                          > messages back to the bus so you don't have to do that.
                                          >
                                          > Currently, there is no way to add/lookup message headers for a
                                          > message right? I believe the MSMQ transport uses a message
                                          envelope,
                                          > but I don't think headers are something that it exposes. It would
                                          be
                                          > nice to have this functionality.
                                          >
                                          > --- In nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> , "Udi Dahan"
                                          > <thesoftwaresimplist@> wrote:
                                          > >
                                          > > That’s an interesting thought â€" using nServiceBus
                                          under the
                                          > ITransport level to implement that functionality. It would break
                                          > configuration based on type, though.
                                          > >
                                          > >
                                          > >
                                          > > --
                                          > >
                                          > > Udi Dahan - The Software Simplist
                                          > >
                                          > >
                                          > >
                                          > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com>
                                          > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> ] On Behalf Of Eric Hauser
                                          > > Sent: 22 May 2008 03:52
                                          > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com>
                                          > > Subject: [nservicebus] Re: Handling failures
                                          > >
                                          > >
                                          > >
                                          > > I don't understand why retry handling would need to be part of
                                          the
                                          > > framework? If you wanted to do that, couldn't you just do
                                          something
                                          > > like (prototype code):
                                          > >
                                          > > public interface IReplayableMessage : IMessage
                                          > > {
                                          > > int RetryCount { get; set; }
                                          > > DateTime LastReplayAttempt { get; set; }
                                          > > }
                                          > >
                                          > > public abstract class RetryMessageHandler<TMessage> :
                                          > > IMessageHandler<TMessage>
                                          > > where TMessage : IReplayableMessage
                                          > > {
                                          > > public IBus Bus { get; set; }
                                          > >
                                          > > public void HandleMessage(TMessage message)
                                          > > {
                                          > > if (DateTime.Now - message.LastReplayAttempt > 5.Seconds())
                                          > > {
                                          > > Bus.HandleMessageLater();
                                          > > return;
                                          > > }
                                          > > try
                                          > > {
                                          > > HandleMessageInternal(message);
                                          > > }
                                          > > catch (Exception e)
                                          > > {
                                          > > if (message.RetryCount < 5)
                                          > > {
                                          > > message.RetryCount = message.RetryCount++;
                                          > > message.LastReplayAttempt = DateTime.Now;
                                          > > Bus.Send(message);
                                          > > }
                                          > > else
                                          > > {
                                          > > .. log error ..
                                          > > }
                                          > > }
                                          > > }
                                          > >
                                          > > public abstract void HandleMessageInternal(TMessage message)
                                          > > }
                                          > >
                                          > > --- In nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com> , "Udi Dahan"
                                          > > <thesoftwaresimplist@> wrote:
                                          > > >
                                          > > > In the example I was talking about, what do you view as the
                                          > > external system? Or does the fact that I assume MSMQ
                                          communication
                                          > > lead you to that conclusion?
                                          > > >
                                          > > >
                                          > > >
                                          > > > --
                                          > > >
                                          > > > Udi Dahan - The Software Simplist
                                          > > >
                                          > > >
                                          > > >
                                          > > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com> ] On Behalf Of pawel
                                          > > > Sent: 21 May 2008 16:14
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > >
                                          > > >
                                          > > > I see your point and it makes sense as long as you control
                                          > external
                                          > > systems. If you don't and they are a bit flaky than a good
                                          > > > retry strategy might help a lot.
                                          > > >
                                          > > > thanks again
                                          > > >
                                          > > > Pawel
                                          > > >
                                          > > > _____
                                          > > >
                                          > > > From: "Udi Dahan" <thesoftwaresimplist@>
                                          > > > Sent: 21 May 2008 13:12
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > > Actually, when you send a message, it first goes to the
                                          outgoing
                                          > > queue on the sending machine. Once that successfully occurs, the
                                          > > sending thread is released to go about its business. Then MSMQ
                                          > tries
                                          > > to get the message to the specified queue on the target machine.
                                          > Once
                                          > > that occurs, the message is deleted from the outgoing queue of
                                          the
                                          > > sending machine. From that point on, the message is sitting in a
                                          > > queue waiting to be processed. When the target process retrieves
                                          > the
                                          > > message from that queue (its input queue), it processes it.
                                          > > >
                                          > > >
                                          > > >
                                          > > > When the DB detects a deadlock between two threads, it kicks
                                          one
                                          > > out, and lets the other proceed. Given that transactions are
                                          > limited
                                          > > in scope to a single message, it is rare that the same thread
                                          will
                                          > > hit a deadlock on the same data when it tries N
                                          micro/milliseconds
                                          > > later, and, even if so, half that chance that it will be selected
                                          > as
                                          > > the victim again. The N micros/millis is the time it takes to
                                          > > rollback the transaction and try again.
                                          > > >
                                          > > >
                                          > > >
                                          > > > That being said, you should be setting the appropriate number
                                          of
                                          > > threads in your autonomous components so that you get the maximal
                                          > > throughput while maintaining minimal contention. The more threads
                                          > are
                                          > > NOT the better. Also, poor DB design can increase the chance of
                                          > > contention, as can processing large messages where transactions
                                          are
                                          > > open longer. This is all applicative design, and has little to do
                                          > > with a retry strategy.
                                          > > >
                                          > > >
                                          > > >
                                          > > > While I can’t categorically say that a retry
                                          strategy isnâ
                                          > €™t
                                          > > needed, I can say that I’ve handled very, VERY
                                          heavy loads
                                          > without
                                          > > needing one, just by profiling, setting the appropriate number of
                                          > > threads, and â€Å"just” by decreasing
                                          message size (as my
                                          > InfoQ
                                          > > article explained).
                                          > > >
                                          > > >
                                          > > >
                                          > > > Hope that explains the reasoning a bit.
                                          > > >
                                          > > >
                                          > > >
                                          > > > --
                                          > > >
                                          > > > Udi Dahan - The Software Simplist
                                          > > >
                                          > > >
                                          > > >
                                          > > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com> ] On Behalf Of pawel
                                          > > > Sent: 21 May 2008 13:16
                                          > > > To: nservicebus@
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > >
                                          > > >
                                          > > > Hi,
                                          > > >
                                          > > > If I understand correctly then whenever a message is sent using
                                          > > MSMQ we deal with 3 distinct steps.
                                          > > >
                                          > > > 1. The message is sent to the sender local queue.
                                          > > > 2. The message is transported from sender local queue to
                                          > recipient
                                          > > local queue.
                                          > > > 3. The message is retrieved from the recipient local queue and
                                          > the
                                          > > recipient processes the message.
                                          > > >
                                          > > > The re-try strategy that I'm talking about is related to the
                                          3rd
                                          > > step. If a database is under heavy load then re-trying
                                          > > > 5 times in a row (without waiting between attempts) is not
                                          going
                                          > to
                                          > > help and MSMQ itself does not provide any
                                          > > > support at this stage(except transactional processing). I was
                                          > > wondering if there is a more sophisticated built-in strategy
                                          > > > that deals with failures when the message is actually being
                                          > > processed by the recipient.
                                          > > >
                                          > > > thanks
                                          > > >
                                          > > > Pawel
                                          > > >
                                          > > > _____
                                          > > >
                                          > > > From: "Udi Dahan" <thesoftwaresimplist@>
                                          > > > Sent: 21 May 2008 08:53
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > > My answer was to say that nServiceBus currently doesnââ
                                          ‚¬â„¢t do
                                          > those
                                          > > kinds of retry strategies â€" MSMQ does that by itself.
                                          > > >
                                          > > >
                                          > > >
                                          > > > That’s one example as to a case where a
                                          different
                                          > transport
                                          > > implementation would expose a different extensibility model.
                                          > > >
                                          > > >
                                          > > >
                                          > > > --
                                          > > >
                                          > > > Udi Dahan - The Software Simplist
                                          > > >
                                          > > >
                                          > > >
                                          > > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com> ] On Behalf Of Andreas ÃÆ'â€"hlund
                                          > > > Sent: 21 May 2008 09:59
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > >
                                          > > >
                                          > > > Isn't Pawel asking about some kind of "Retry strategy". The
                                          > current
                                          > > one is "Retry 5 time with no wait". I've seen solutions that
                                          sleeps
                                          > a
                                          > > while before retrying ( and perhaps extends the "sleep period"
                                          > based
                                          > > on number of failed retries)
                                          > > >
                                          > > > It would also be nice if the bus or transport could expose
                                          > > an "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I
                                          know
                                          > > you could read the error queue but this would be a more transport
                                          > > agnostic solution?)
                                          > > >
                                          > > > Thoughts?
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > > _____
                                          > > >
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > From: thesoftwaresimplist@
                                          > > > Date: Wed, 21 May 2008 01:27:43 +0300
                                          > > > Subject: RE: [nservicebus] Handling failures
                                          > > >
                                          > > > The retry count doesn’t have to do with
                                          message delivery.
                                          > MSMQ
                                          > > handles that independently of nServiceBus.
                                          > > >
                                          > > >
                                          > > >
                                          > > > When a transaction is thrown during any part of processing the
                                          > > message, then we retry the HANDLING of the message.
                                          Deserialization
                                          > > exceptions are different â€" those are immediately
                                          transferred
                                          > to the
                                          > > error queue without any retries.
                                          > > >
                                          > > >
                                          > > >
                                          > > > The reason for this logic has to do with technological issues
                                          > > preventing otherwise successful message handling â€"
                                          deadlocks
                                          > in the
                                          > > database are one example.
                                          > > >
                                          > > >
                                          > > >
                                          > > > Does that answer your question?
                                          > > >
                                          > > >
                                          > > >
                                          > > > --
                                          > > >
                                          > > > Udi Dahan - The Software Simplist
                                          > > >
                                          > > >
                                          > > >
                                          > > > From: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > [mailto:nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com> ] On Behalf Of pawel.pabich
                                          > > > Sent: 21 May 2008 00:48
                                          > > > To: nservicebus@yahoogroups.com <mailto:nservicebus%
                                          40yahoogroups.com> <mailto:nservicebus%
                                          > 40yahoogroups.com>
                                          > > > Subject: [nservicebus] Handling failures
                                          > > >
                                          > > >
                                          > > >
                                          > > > As far as I know if a message delivery fails then NServiceBus
                                          > > retries
                                          > > > immediately and keeps doing this until 5th attempt in a row
                                          > fails.
                                          > > If
                                          > > > that happens then it sends the message to the error queue. Is
                                          > there
                                          > > > any setting that would instruct NServiceBus to wait some period
                                          of
                                          > > > time between retries? Is there any other built-in way of
                                          handling
                                          > > such
                                          > > > cases when the first delivery fails?
                                          > > >
                                          > > > thanks
                                          > > >
                                          > > > Pawel
                                          > > >
                                          > > > No virus found in this incoming message.
                                          > > > Checked by AVG.
                                          > > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release
                                          Date:
                                          > > 20/05/2008 16:45
                                          > > >
                                          > > >
                                          > > >
                                          > > > _____
                                          > > >
                                          > > > Den perfekta mixen av nÃÆ'¶jesnyheter livekonserter! MSN
                                          Video
                                          > > <http://video.se.msn.com/v/sv-se/v.htm>
                                          > > >
                                          > > > No virus found in this incoming message.
                                          > > > Checked by AVG.
                                          > > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release
                                          Date:
                                          > > 20/05/2008 16:45
                                          > > >
                                          > > >
                                          > > >
                                          > > > No virus found in this incoming message.
                                          > > > Checked by AVG.
                                          > > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release
                                          Date:
                                          > > 20/05/2008 16:45
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > > No virus found in this incoming message.
                                          > > > Checked by AVG.
                                          > > > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release
                                          Date:
                                          > > 20/05/2008 16:45
                                          > > >
                                          > >
                                          > >
                                          > >
                                          > > No virus found in this incoming message.
                                          > > Checked by AVG.
                                          > > Version: 8.0.100 / Virus Database: 269.23.21/1458 - Release Date:
                                          > 21/05/2008 07:21
                                          > >
                                          >
                                          >
                                          >
                                          > No virus found in this incoming message.
                                          > Checked by AVG.
                                          > Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date:
                                          21/05/2008 17:34
                                          >
                                        • Pawel Pabich
                                          Putting retry strategy at the level of infrastructure doesn t mean that I want to keep the number of retires in memory. It can be serialized together with the
                                          Message 20 of 20 , May 22, 2008
                                          View Source
                                          • 0 Attachment
                                            Putting retry strategy at the level of infrastructure doesn't mean that
                                            I want to keep
                                            the number of retires in memory. It can be serialized together with the
                                            message but
                                            the application shouldn't be aware of that. The property bag that Erik
                                            suggests might be an option.

                                            I agree that indirection helps keep things simple but on the other hand
                                            too many indirections
                                            can make life much harder. I suppose everyone tries to strike the right
                                            balance.

                                            Pawel

                                            > Just keep in mind what happens to that state (the number of retries)
                                            > when your server restarts. Maybe the answer is “I don’t care, I’ll
                                            > just start from scratch”, but there are some protocols where you do
                                            > need to know at which stage you failed, and in what way.
                                            >
                                            >
                                            >
                                            > One last thing, indirection helps move complexity around, a core
                                            > capability in keeping your system testable and maintainable.
                                            >
                                            >
                                            >
                                            > Do you have a specific feature request in mind?
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *pawel
                                            > *Sent:* 22 May 2008 17:15
                                            > *To:* nservicebus@yahoogroups.com
                                            > *Subject:* RE: [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > From my perspective this is like adding another level of indirection.
                                            > At the end of the day there must be a piece of code that handles
                                            > retries and
                                            > it does not really matter if it lives inside the message handler or in
                                            > a separate component.
                                            > By keeping the retry logic inside the message handler I can piggyback
                                            > on the existing ambient transaction and MSMQ as
                                            > the reliable storage.
                                            >
                                            > Pawel
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > *From*: "Udi Dahan" <thesoftwaresimplist@...>
                                            > *Sent*: 22 May 2008 13:25
                                            > *To*: nservicebus@yahoogroups.com
                                            > *Subject*: RE: [nservicebus] Handling failures
                                            >
                                            > Now I understand what your scenario is!
                                            >
                                            >
                                            >
                                            > The way that external integration of that kind is handled is simple:
                                            >
                                            >
                                            >
                                            > Your application logic sits in one autonomous component. Another
                                            > autonomous component serves as an anti-corruption layer, actually
                                            > communicating with the external system. Your logic sends a message to
                                            > the other AC. That AC handles the message and calls the synchronous
                                            > service. Should there be a problem there, an exception will cause a
                                            > retry. You can define the number of retries for that AC separately
                                            > from everything else.
                                            >
                                            >
                                            >
                                            > If you want to inject a time component to the process, use sagas and
                                            > the timeout manager.
                                            >
                                            >
                                            >
                                            > Does that help?
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *pawel
                                            > *Sent:* 22 May 2008 13:23
                                            > *To:* nservicebus@yahoogroups.com
                                            > *Subject:* RE: [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > By the external system I mean a system that the service needs to talk
                                            > to synchronously when it processes a message.
                                            > As an example, there is a system that uses a home-grown database as
                                            > the back-end which leads to all sort of unexpected behaviours.
                                            >
                                            > thanks
                                            >
                                            > Pawel
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > *From*: "Udi Dahan" <thesoftwaresimplist@...>
                                            > *Sent*: 21 May 2008 14:52
                                            > *To*: nservicebus@yahoogroups.com
                                            > *Subject*: RE: [nservicebus] Handling failures
                                            >
                                            > In the example I was talking about, what do you view as the external
                                            > system? Or does the fact that I assume MSMQ communication lead you to
                                            > that conclusion?
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *pawel
                                            > *Sent:* 21 May 2008 16:14
                                            > *To:* nservicebus@yahoogroups.com
                                            > *Subject:* RE: [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > I see your point and it makes sense as long as you control external
                                            > systems. If you don't and they are a bit flaky than a good
                                            > retry strategy might help a lot.
                                            >
                                            > thanks again
                                            >
                                            > Pawel
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > *From*: "Udi Dahan" <thesoftwaresimplist@...>
                                            > *Sent*: 21 May 2008 13:12
                                            > *To*: nservicebus@yahoogroups.com
                                            > *Subject*: RE: [nservicebus] Handling failures
                                            >
                                            > Actually, when you send a message, it first goes to the outgoing queue
                                            > on the sending machine. Once that successfully occurs, the sending
                                            > thread is released to go about its business. Then MSMQ tries to get
                                            > the message to the specified queue on the target machine. Once that
                                            > occurs, the message is deleted from the outgoing queue of the sending
                                            > machine. From that point on, the message is sitting in a queue waiting
                                            > to be processed. When the target process retrieves the message from
                                            > that queue (its input queue), it processes it.
                                            >
                                            >
                                            >
                                            > When the DB detects a deadlock between two threads, it kicks one out,
                                            > and lets the other proceed. Given that transactions are limited in
                                            > scope to a single message, it is rare that the same thread will hit a
                                            > deadlock on the same data when it tries N micro/milliseconds later,
                                            > and, even if so, half that chance that it will be selected as the
                                            > victim again. The N micros/millis is the time it takes to rollback the
                                            > transaction and try again.
                                            >
                                            >
                                            >
                                            > That being said, you should be setting the appropriate number of
                                            > threads in your autonomous components so that you get the maximal
                                            > throughput while maintaining minimal contention. The more threads are
                                            > NOT the better. Also, poor DB design can increase the chance of
                                            > contention, as can processing large messages where transactions are
                                            > open longer. This is all applicative design, and has little to do with
                                            > a retry strategy.
                                            >
                                            >
                                            >
                                            > While I can’t categorically say that a retry strategy isn’t needed, I
                                            > can say that I’ve handled very, VERY heavy loads without needing one,
                                            > just by profiling, setting the appropriate number of threads, and
                                            > “just” by decreasing message size (as my InfoQ article explained).
                                            >
                                            >
                                            >
                                            > Hope that explains the reasoning a bit.
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *pawel
                                            > *Sent:* 21 May 2008 13:16
                                            > *To:* nservicebus@yahoogroups..com
                                            > *Subject:* RE: [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > Hi,
                                            >
                                            > If I understand correctly then whenever a message is sent using MSMQ
                                            > we deal with 3 distinct steps.
                                            >
                                            > 1. The message is sent to the sender local queue.
                                            > 2. The message is transported from sender local queue to recipient
                                            > local queue.
                                            > 3. The message is retrieved from the recipient local queue and the
                                            > recipient processes the message.
                                            >
                                            > The re-try strategy that I'm talking about is related to the 3rd step.
                                            > If a database is under heavy load then re-trying
                                            > 5 times in a row (without waiting between attempts) is not going to
                                            > help and MSMQ itself does not provide any
                                            > support at this stage(except transactional processing). I was
                                            > wondering if there is a more sophisticated built-in strategy
                                            > that deals with failures when the message is actually being processed
                                            > by the recipient.
                                            >
                                            > thanks
                                            >
                                            > Pawel
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > *From*: "Udi Dahan" <thesoftwaresimplist@...>
                                            > *Sent*: 21 May 2008 08:53
                                            > *To*: nservicebus@yahoogroups.com
                                            > *Subject*: RE: [nservicebus] Handling failures
                                            >
                                            > My answer was to say that nServiceBus currently doesn’t do those kinds
                                            > of retry strategies – MSMQ does that by itself.
                                            >
                                            >
                                            >
                                            > That’s one example as to a case where a different transport
                                            > implementation would expose a different extensibility model.
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *Andreas Öhlund
                                            > *Sent:* 21 May 2008 09:59
                                            > *To:* nservicebus@yahoogroups.com
                                            > *Subject:* RE: [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > Isn't Pawel asking about some kind of "Retry strategy". The current
                                            > one is "Retry 5 time with no wait". I've seen solutions that sleeps a
                                            > while before retrying ( and perhaps extends the "sleep period" based
                                            > on number of failed retries)
                                            >
                                            > It would also be nice if the bus or transport could expose an
                                            > "MessageRetriedEvent" and a "MessageHandlingFailedEvent" (I know you
                                            > could read the error queue but this would be a more transport agnostic
                                            > solution?)
                                            >
                                            > Thoughts?
                                            >
                                            >
                                            >
                                            >
                                            >
                                            >
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > To: nservicebus@yahoogroups.com
                                            > From: thesoftwaresimplist@...
                                            > Date: Wed, 21 May 2008 01:27:43 +0300
                                            > Subject: RE: [nservicebus] Handling failures
                                            >
                                            > The retry count doesn’t have to do with message delivery. MSMQ
                                            > handles that independently of nServiceBus.
                                            >
                                            >
                                            >
                                            > When a transaction is thrown during any part of processing the
                                            > message, then we retry the HANDLING of the message.
                                            > Deserialization exceptions are different – those are immediately
                                            > transferred to the error queue without any retries.
                                            >
                                            >
                                            >
                                            > The reason for this logic has to do with technological issues
                                            > preventing otherwise successful message handling – deadlocks in
                                            > the database are one example.
                                            >
                                            >
                                            >
                                            > Does that answer your question?
                                            >
                                            >
                                            >
                                            > --
                                            >
                                            > Udi Dahan - The Software Simplist
                                            >
                                            >
                                            >
                                            > *From:* nservicebus@yahoogroups.com
                                            > [mailto:nservicebus@yahoogroups.com] *On Behalf Of *pawel.pabich
                                            > *Sent:* 21 May 2008 00:48
                                            > *To:* nservicebus@yahoogroups.com
                                            > *Subject:* [nservicebus] Handling failures
                                            >
                                            >
                                            >
                                            > As far as I know if a message delivery fails then NServiceBus retries
                                            > immediately and keeps doing this until 5th attempt in a row fails. If
                                            > that happens then it sends the message to the error queue. Is there
                                            > any setting that would instruct NServiceBus to wait some period of
                                            > time between retries? Is there any other built-in way of handling such
                                            > cases when the first delivery fails?
                                            >
                                            > thanks
                                            >
                                            > Pawel
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                            > 20/05/2008 16:45
                                            >
                                            >
                                            >
                                            > ------------------------------------------------------------------------
                                            >
                                            > Den perfekta mixen av nöjesnyheter livekonserter! MSN Video
                                            > <http://video.se.msn.com/v/sv-se/v.htm>
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                            > 20/05/2008 16:45
                                            >
                                            >
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                            > 20/05/2008 16:45
                                            >
                                            >
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.23.21/1457 - Release Date:
                                            > 20/05/2008 16:45
                                            >
                                            >
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date:
                                            > 21/05/2008 17:34
                                            >
                                            >
                                            >
                                            > No virus found in this incoming message.
                                            > Checked by AVG.
                                            > Version: 8.0.100 / Virus Database: 269.24.0/1459 - Release Date:
                                            > 21/05/2008 17:34
                                            >
                                            >
                                          Your message has been successfully submitted and would be delivered to recipients shortly.