Loading ...
Sorry, an error occurred while loading the content.

Re: [domaindrivendesign] Processing Events in Correct Sequence Order

Expand Messages
  • Greg Young
    Order is only assured per a handler within an aggregate root boundary. There is no assurance of order between handlers or between aggregates. Trying to provide
    Message 1 of 10 , May 10, 2010
    View Source
    • 0 Attachment
      Order is only assured per a handler within an aggregate root boundary.
       
      There is no assurance of order between handlers or between aggregates.
       
      Trying to provide those things leads to the dark side.

      On Thu, May 6, 2010 at 5:10 PM, Hendry Luk <hendrymail@...> wrote:
       

      Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



      On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
       

      It should only matter within an aggregate boundary.

      Sent from my iPhone

      On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

       

      Good day,
      This is becoming less and less about DDD. We probably really need a forum for cqrs...

      I'll ask away anyway...
      I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
      1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
      2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

      So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

      There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

      My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
      I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

      So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

      Cheers in advance guys
      Hendry





      --
      Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention
    • Hendry Luk
      Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same
      Message 2 of 10 , May 14, 2010
      View Source
      • 0 Attachment
        Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same aggregate instance.

        I find it perfectly logical if we were talking about internal event-handler within aggregate-roots. But I'm still struggling to grasp this rule in the context of event-handling within denormaliser and inter-AR services. A denormaliser, by definition, deals with many different aggregate roots. Failure to preserve the correct sequence of events coming from different aggregate roots may result in data inconsistencies.

        E.g. "IT" team moves from Glasgow to Lisbon, and soon afterward "Tom", our beloved hypothetical character, moves from "Sales" (Madrid) to "IT" (now Lisbon), they'are 2 events from 2 different ARs, and if they are processed in opposite sequence, it will look like "Tom" just travelled all around Europe from Madrid to Glasgow then to Lisbon.. while in actual fact he has never actually left anywhere too far from sunlight. This may cause the system to produce wrong EmployeeRellocations report.
        Even worse, if the company creates a new "Finance" team, then "James" joins the team, if these 2 events are processed in the wrong order, you will get a problem because "James" is joining a team that has yet to exist.

        I did a quick search in t'internet, and found that Axon framework has a special topic on this.
        "The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate."
        (http://www.gridshore.nl/2010/02/21/axon-framework-the-cqrs-framework-for-java-version-0-4-released/)

        And by the way, Axon also provides a mechanism to easily reconfigure event-processing between full and eventual consistencies. It's not very easy to find implementation details on these, and I don't use Axon. So would appreciate if anyone can shed a light on the principle or share any thought.

        Cheers
        Hendry


        On Mon, May 10, 2010 at 6:04 PM, bodrin <bodrin@...> wrote:
         

        Hi, there was a similar discussion here
        http://tech.groups.yahoo.com/group/domaindrivendesign/message/12870

        The point is that in 99.9% of the use-cases you do not need a GLOBAL ORDER. You don't have to maintain consistency across aggregate instances (except referential integrity). You need to order only per aggregate instance (consistency boundery) :

        ar.id=X ar.version=1
        ar.id=X ar.version=2
        ar.id=X ar.version=3
        ...




        On Thu, May 6, 2010 at 6:10 PM, Hendry Luk <hendrymail@...> wrote:
         

        Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



        On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
         

        It should only matter within an aggregate boundary.

        Sent from my iPhone

        On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

         

        Good day,
        This is becoming less and less about DDD. We probably really need a forum for cqrs...

        I'll ask away anyway...
        I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
        1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
        2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

        So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

        There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

        My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
        I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

        So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

        Cheers in advance guys
        Hendry




      • Greg Young
        For full sync vs async its just a matter of where you set your transaction boundary (does it include the event handlers?). ... -- Les erreurs de grammaire et
        Message 3 of 10 , May 14, 2010
        View Source
        • 0 Attachment
          For full sync vs async its just a matter of where you set your transaction boundary (does it include the event handlers?).

          On Fri, May 14, 2010 at 4:24 AM, Hendry Luk <hendrymail@...> wrote:
           

          Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same aggregate instance.

          I find it perfectly logical if we were talking about internal event-handler within aggregate-roots. But I'm still struggling to grasp this rule in the context of event-handling within denormaliser and inter-AR services. A denormaliser, by definition, deals with many different aggregate roots. Failure to preserve the correct sequence of events coming from different aggregate roots may result in data inconsistencies.

          E.g. "IT" team moves from Glasgow to Lisbon, and soon afterward "Tom", our beloved hypothetical character, moves from "Sales" (Madrid) to "IT" (now Lisbon), they'are 2 events from 2 different ARs, and if they are processed in opposite sequence, it will look like "Tom" just travelled all around Europe from Madrid to Glasgow then to Lisbon.. while in actual fact he has never actually left anywhere too far from sunlight. This may cause the system to produce wrong EmployeeRellocations report.
          Even worse, if the company creates a new "Finance" team, then "James" joins the team, if these 2 events are processed in the wrong order, you will get a problem because "James" is joining a team that has yet to exist.

          I did a quick search in t'internet, and found that Axon framework has a special topic on this.
          "The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate."
          (http://www.gridshore.nl/2010/02/21/axon-framework-the-cqrs-framework-for-java-version-0-4-released/)

          And by the way, Axon also provides a mechanism to easily reconfigure event-processing between full and eventual consistencies. It's not very easy to find implementation details on these, and I don't use Axon. So would appreciate if anyone can shed a light on the principle or share any thought.

          Cheers
          Hendry




          On Mon, May 10, 2010 at 6:04 PM, bodrin <bodrin@...> wrote:
           

          Hi, there was a similar discussion here
          http://tech.groups.yahoo.com/group/domaindrivendesign/message/12870

          The point is that in 99.9% of the use-cases you do not need a GLOBAL ORDER. You don't have to maintain consistency across aggregate instances (except referential integrity). You need to order only per aggregate instance (consistency boundery) :

          ar.id=X ar.version=1
          ar.id=X ar.version=2
          ar.id=X ar.version=3
          ...




          On Thu, May 6, 2010 at 6:10 PM, Hendry Luk <hendrymail@...> wrote:
           

          Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



          On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
           

          It should only matter within an aggregate boundary.

          Sent from my iPhone

          On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

           

          Good day,
          This is becoming less and less about DDD. We probably really need a forum for cqrs...

          I'll ask away anyway...
          I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
          1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
          2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

          So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

          There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

          My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
          I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

          So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

          Cheers in advance guys
          Hendry







          --
          Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention
        • bodrin
          hi, ... 1. CreateTeam( Finance ) - results in teamId=5 2. JoinEmployeeToTeam(employId=3, teamId=5) If the denormaliser gets these in reverse order: 1.
          Message 4 of 10 , May 18, 2010
          View Source
          • 0 Attachment
            hi,

            On Fri, May 14, 2010 at 11:24 AM, Hendry Luk <hendrymail@...> wrote:
             

            Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same aggregate instance.

            I find it perfectly logical if we were talking about internal event-handler within aggregate-roots. But I'm still struggling to grasp this rule in the context of event-handling within denormaliser and inter-AR services. A denormaliser, by definition, deals with many different aggregate roots. Failure to preserve the correct sequence of events coming from different aggregate roots may result in data inconsistencies.

            E.g. "IT" team moves from Glasgow to Lisbon, and soon afterward "Tom", our beloved hypothetical character, moves from "Sales" (Madrid) to "IT" (now Lisbon), they'are 2 events from 2 different ARs, and if they are processed in opposite sequence, it will look like "Tom" just travelled all around Europe from Madrid to Glasgow then to Lisbon.. while in actual fact he has never actually left anywhere too far from sunlight. This may cause the system to produce wrong EmployeeRellocations report.
            Even worse, if the company creates a new "Finance" team, then "James" joins the team, if these 2 events are processed in the wrong order, you will get a problem because "James" is joining a team that has yet to exist.


            1. CreateTeam("Finance") -> results in teamId=5
            2. JoinEmployeeToTeam(employId=3, teamId=5)

            If the denormaliser gets these in reverse order:

            1. JoinEmployeeToTeam(employId=3, teamId=5)
            2. CreateTeam("Finance") -> teamId=5

            ..it can postpone the processing of JoinEmployeeToTeam event, because it does not see team with id=5 and thus it performs a kind of ordering which prevents the referential integrity violation *only*. So when it sees this event it will wait until it sees Team AR with id=5 and then it will be eligible for processing.

            some other thoughts:

            Other *not so critical* events could be ordered by the denormalizer by timestamp. For example assuming that an event could overdue not more then 10 minutes passing from a domain node to the denormalizer and the system clock differences between any two domain nodes is not more then 100ms. This way the denormalizer will be always at least 10 minutes back from the domain and you could not order events which differ in timestamp less then 100ms.

            Also if you read the post that I refer above you will find that you can denormalize things that you want to be strong consistent with into the event as Greg advised. This way the denormalizer will be able to do the right calculations. This however could be not very practical in some cases..


            I did a quick search in t'internet, and found that Axon framework has a special topic on this.
            "The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate."
            (http://www.gridshore.nl/2010/02/21/axon-framework-the-cqrs-framework-for-java-version-0-4-released/)

            And by the way, Axon also provides a mechanism to easily reconfigure event-processing between full and eventual consistencies. It's not very easy to find implementation details on these, and I don't use Axon. So would appreciate if anyone can shed a light on the principle or share any thought.

            Cheers
            Hendry




            On Mon, May 10, 2010 at 6:04 PM, bodrin <bodrin@...> wrote:
             

            Hi, there was a similar discussion here
            http://tech.groups.yahoo.com/group/domaindrivendesign/message/12870

            The point is that in 99.9% of the use-cases you do not need a GLOBAL ORDER. You don't have to maintain consistency across aggregate instances (except referential integrity). You need to order only per aggregate instance (consistency boundery) :

            ar.id=X ar.version=1
            ar.id=X ar.version=2
            ar.id=X ar.version=3
            ...




            On Thu, May 6, 2010 at 6:10 PM, Hendry Luk <hendrymail@...> wrote:
             

            Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



            On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
             

            It should only matter within an aggregate boundary.

            Sent from my iPhone

            On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

             

            Good day,
            This is becoming less and less about DDD. We probably really need a forum for cqrs...

            I'll ask away anyway...
            I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
            1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
            2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

            So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

            There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

            My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
            I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

            So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

            Cheers in advance guys
            Hendry





          • Hendry Luk
            Actually sync/async in Axon is defined for each individual event-handler, defining where the event-processing should take place, i.e., within same thread
            Message 5 of 10 , May 20, 2010
            View Source
            • 0 Attachment
              Actually sync/async in Axon is defined for each individual event-handler, defining where the event-processing should take place, i.e., within same thread (default), or asynchronously from a different process. This is separate from the sequencing issue.

              If you choose to use async event-handler, apparently you'll need to also specify the SequencingPolicy for that particular event-handler, which allows you to define whether it should process the events in a perfect sequence, or completely parallel, or sequential only within aggregate boundary.

              On Fri, May 14, 2010 at 10:41 PM, Greg Young <gregoryyoung1@...> wrote:
               

              For full sync vs async its just a matter of where you set your transaction boundary (does it include the event handlers?).



              On Fri, May 14, 2010 at 4:24 AM, Hendry Luk <hendrymail@...> wrote:
               

              Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same aggregate instance.

              I find it perfectly logical if we were talking about internal event-handler within aggregate-roots. But I'm still struggling to grasp this rule in the context of event-handling within denormaliser and inter-AR services. A denormaliser, by definition, deals with many different aggregate roots. Failure to preserve the correct sequence of events coming from different aggregate roots may result in data inconsistencies.

              E.g. "IT" team moves from Glasgow to Lisbon, and soon afterward "Tom", our beloved hypothetical character, moves from "Sales" (Madrid) to "IT" (now Lisbon), they'are 2 events from 2 different ARs, and if they are processed in opposite sequence, it will look like "Tom" just travelled all around Europe from Madrid to Glasgow then to Lisbon.. while in actual fact he has never actually left anywhere too far from sunlight. This may cause the system to produce wrong EmployeeRellocations report.
              Even worse, if the company creates a new "Finance" team, then "James" joins the team, if these 2 events are processed in the wrong order, you will get a problem because "James" is joining a team that has yet to exist.

              I did a quick search in t'internet, and found that Axon framework has a special topic on this.
              "The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate."
              (http://www.gridshore.nl/2010/02/21/axon-framework-the-cqrs-framework-for-java-version-0-4-released/)

              And by the way, Axon also provides a mechanism to easily reconfigure event-processing between full and eventual consistencies. It's not very easy to find implementation details on these, and I don't use Axon. So would appreciate if anyone can shed a light on the principle or share any thought.

              Cheers
              Hendry




              On Mon, May 10, 2010 at 6:04 PM, bodrin <bodrin@...> wrote:
               

              Hi, there was a similar discussion here
              http://tech.groups.yahoo.com/group/domaindrivendesign/message/12870

              The point is that in 99.9% of the use-cases you do not need a GLOBAL ORDER. You don't have to maintain consistency across aggregate instances (except referential integrity). You need to order only per aggregate instance (consistency boundery) :

              ar.id=X ar.version=1
              ar.id=X ar.version=2
              ar.id=X ar.version=3
              ...




              On Thu, May 6, 2010 at 6:10 PM, Hendry Luk <hendrymail@...> wrote:
               

              Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



              On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
               

              It should only matter within an aggregate boundary.

              Sent from my iPhone

              On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

               

              Good day,
              This is becoming less and less about DDD. We probably really need a forum for cqrs...

              I'll ask away anyway...
              I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
              1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
              2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

              So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

              There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

              My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
              I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

              So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

              Cheers in advance guys
              Hendry







              --
              Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention

            • Greg Young
              yes thats what I was saying, the sync/async is only where you set the transaction boundary. ... -- Les erreurs de grammaire et de syntaxe ont été incluses
              Message 6 of 10 , May 21, 2010
              View Source
              • 0 Attachment
                yes thats what I was saying, the sync/async is only where you set the transaction boundary.

                On Fri, May 21, 2010 at 12:53 AM, Hendry Luk <hendrymail@...> wrote:
                 

                Actually sync/async in Axon is defined for each individual event-handler, defining where the event-processing should take place, i.e., within same thread (default), or asynchronously from a different process. This is separate from the sequencing issue.

                If you choose to use async event-handler, apparently you'll need to also specify the SequencingPolicy for that particular event-handler, which allows you to define whether it should process the events in a perfect sequence, or completely parallel, or sequential only within aggregate boundary.

                On Fri, May 14, 2010 at 10:41 PM, Greg Young <gregoryyoung1@...> wrote:
                 

                For full sync vs async its just a matter of where you set your transaction boundary (does it include the event handlers?).



                On Fri, May 14, 2010 at 4:24 AM, Hendry Luk <hendrymail@...> wrote:
                 

                Thanks Greg and Bodrin. You both seem to be in agreeement on advising that we should only guarantee the sequential processing of events that come from the same aggregate instance.

                I find it perfectly logical if we were talking about internal event-handler within aggregate-roots. But I'm still struggling to grasp this rule in the context of event-handling within denormaliser and inter-AR services. A denormaliser, by definition, deals with many different aggregate roots. Failure to preserve the correct sequence of events coming from different aggregate roots may result in data inconsistencies.

                E.g. "IT" team moves from Glasgow to Lisbon, and soon afterward "Tom", our beloved hypothetical character, moves from "Sales" (Madrid) to "IT" (now Lisbon), they'are 2 events from 2 different ARs, and if they are processed in opposite sequence, it will look like "Tom" just travelled all around Europe from Madrid to Glasgow then to Lisbon.. while in actual fact he has never actually left anywhere too far from sunlight. This may cause the system to produce wrong EmployeeRellocations report.
                Even worse, if the company creates a new "Finance" team, then "James" joins the team, if these 2 events are processed in the wrong order, you will get a problem because "James" is joining a team that has yet to exist.

                I did a quick search in t'internet, and found that Axon framework has a special topic on this.
                "The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate."
                (http://www.gridshore.nl/2010/02/21/axon-framework-the-cqrs-framework-for-java-version-0-4-released/)

                And by the way, Axon also provides a mechanism to easily reconfigure event-processing between full and eventual consistencies. It's not very easy to find implementation details on these, and I don't use Axon. So would appreciate if anyone can shed a light on the principle or share any thought.

                Cheers
                Hendry




                On Mon, May 10, 2010 at 6:04 PM, bodrin <bodrin@...> wrote:
                 

                Hi, there was a similar discussion here
                http://tech.groups.yahoo.com/group/domaindrivendesign/message/12870

                The point is that in 99.9% of the use-cases you do not need a GLOBAL ORDER. You don't have to maintain consistency across aggregate instances (except referential integrity). You need to order only per aggregate instance (consistency boundery) :

                ar.id=X ar.version=1
                ar.id=X ar.version=2
                ar.id=X ar.version=3
                ...




                On Thu, May 6, 2010 at 6:10 PM, Hendry Luk <hendrymail@...> wrote:
                 

                Sorry greg, I'm not 100% clear whether you were meaning: "events from 1 aggregate root need to be processed by all event handlers in that exact order", or simply "events only need to be in correct order when being loaded within the aggregate-root itself (to rebuild the object)"?



                On Fri, May 7, 2010 at 12:48 AM, Greg Young <gregoryyoung1@...> wrote:
                 

                It should only matter within an aggregate boundary.

                Sent from my iPhone

                On May 6, 2010, at 16:40, Hendry Luk <hendrymail@...> wrote:

                 

                Good day,
                This is becoming less and less about DDD. We probably really need a forum for cqrs...

                I'll ask away anyway...
                I'm planning to have 2 different event-sourcing mechanisms in one application for different scenarios. Some event-handlers use messaging, other scenarios might call for good old synchronous handlers. I want to be able to configure each event-handler to tell which event bus to subscribe:
                1. Direct-bus, events will be processed synchronously within the same process to avoid the complexity of eventual-consistency in user interaction
                2. Messaging-bus, events will be processed asynchronously to gain scalability benefit

                So each event from unit-of-work will be published to 2 buses. Each event-handler can subscribe to either one of these buses. Some event-handler might start with direct-bus for simplicity and improved user experience (in certain scenarios that call for it), and might get switched later to message-bus when performance starts becoming an issue in a case per case basis.

                There's however a problem of publishing events through 2 different buses. Even though I guarantee that each event-handler processes all events in the correct order, which is very important, however across different event-handlers, it may not be the case. Sync-handlers will obviously always receive their events earlier that messaging ones do.

                My question is... how important do you think it is to have all events to get processed across all event-handlers in exact order? I know the dodgy-ness involved, but how serious? The thing is, I couldn't think of any alternative.
                I like the simplicity of immediate consistency, and the scalability of messaging, but I dont want to be limited to pick only either one of them as a one-size-fit-all solution and apply it equally to every part of the application. Not all parts are the same.. there are some scenarios where messaging may not be your dearest friend. But I sense that what I'm doing is sinfully dangerous... messing up the order in which events are processed.

                So before I commit to walk down this route, I was hoping to hear your thoughts about having some of your event handlers receive events earlier than the others. Or if you think I'm doing something utterly idiotic. Have you attempted to have both messaging and synchronous solutions for different scenarios within one application? Any better way of doing it?

                Cheers in advance guys
                Hendry







                --
                Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention




                --
                Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention
              Your message has been successfully submitted and would be delivered to recipients shortly.