Loading ...
Sorry, an error occurred while loading the content.

Re: [nservicebus] Working on Amazon SQS transport

Expand Messages
  • Ayende Rahien
    Another issue is that a _lot_ of the assumptions that NSB makes about things like rolling back a message, or whatever the queue can throw errors or not
    Message 1 of 8 , Mar 14, 2009
    View Source
    • 0 Attachment
      Another issue is that a _lot_ of the assumptions that NSB makes about things like rolling back a message, or whatever the queue can throw errors or not

      On Sat, Mar 14, 2009 at 7:35 AM, Udi Dahan <thesoftwaresimplist@...> wrote:

      Erik,

       

      While it does make sense to use MSMQ as a subscription storage implementation, Amazon SQS much less so. S3 would be a better choice.

       

      Hope that helps,

       

      -- Udi

       

       

      From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Erik Westermann
      Sent: Thursday, March 05, 2009 1:21 AM
      To: nservicebus@yahoogroups.com
      Subject: [nservicebus] Working on Amazon SQS transport

       

      I'm working on adding to Amazon's SQS as a new transport in nServiceBus.
      Amazon SQS (Simple Queue Service - information here:
      http://aws.amazon.com/sqs/).

      This discusses some of the issues I ran into and how I addressed them.
      You might be interested in this if you are considering adding your own
      transport to nServiceBus, or are just curious about some of nServiceBus'
      implementation techniques.

      My motivation for adding a remote, distributed queue to nServiceBus was
      to gain a better understanding of nServiceBus and Amazon SQS. Using a
      distributed, third party transport also offers some novel deployment
      scenarios and capabilities.

      Amazon SQS is relatively easy to use and, fortunately, Amazon makes a
      sample .NET library available to make it easier to work with. The
      samples demonstrate how to create, list, and delete queues and how to
      send and receive messages. Oddly enough, the sample code is not in line
      with certain recommendations for working with SQS queues but once I got
      around that, the services worked as documented.

      nServiceBus assumes connected, low latency communications with the
      underlying storage - MSMQ. For example, when a publisher initializes, it
      checks the subscriptions queue by iterating over all subscription
      messages in the queue. The publisher does this to associate message
      types with subscription queues, making it possible to route messages to
      the right queue. The iteration occurs in a foreach loop based on a call
      to MSMQ's GetAllMessages method.

      Transitioning the default connected model to Amazon SQS was not easy
      since the disconnected nature of SQS needs to be hidden behind a
      programming model that not only simulates a connected environment, but
      also handles some of the SQS's features and capabilities. One of the
      more significant issues is getting the total number of messages in an
      SQS queue - Amazon SQS returns an approximate number of messages since
      the underlying queue is highly distributed. Amazon warns against
      fetching a specific number of messages, recomending instead to keep
      polling for messages until you have a certian number of consecutive
      null/empty receives. I did not have the patience to implement an
      elaborate solution to address these issues, so I used a more basic
      approach that's more likely to result in delayed messages (good enough
      for now).

      Another aspect of the default nServiceBus transport is its tight
      coupling with MSMQ - the coupling and other details are well hidden
      behind the ITransport interface limiting it's impact to code within the
      transport itself.

      An example of the tight coupling is when the nServiceBus MSMQ transport
      uses an MSMQ message's Label and Id properties to handle initialization
      and to determine a message's destination. This approach is efficient and
      probably very fast.

      Personally, I would have created a new type of message and added a bit
      more processing in my code rather than relying on features of the
      underlying transport. Although my approach might end up being a little
      slower and use more code it would work to not only make it easier to add
      new transports, but also provide a greater degree of control over the
      message itself. Using transport-specific attributes is analogous to
      adding context to a message and representing that context using the
      transport's capabilities. nSerivceBus' approach works well, I am just
      mentioning it for others that might be interested in adding their own
      transports.

      Configuration in nServiceBus is tricky - fortunately the included MSMQ
      transport is easy to use as a model since it's easy to follow.

      Once things were working, I managed to send and receive messages using
      the PubSub sample; however, the Server did not last long - it crashed
      pretty quickly. My Server is crashing because my replacement for the
      MSMQ Id and Label properties (and some other MSMQ-related code) is still
      under development. The Subscriber, after a few tries, ran perfectly.

      Overall, my experience with nServiceBus 1.9 is positive and I am looking
      forward to updates to the container model that's been mentioned here on
      the list during the past few days. Amazon SQS is very robust and offers
      a lot of features for the tiny price that they charge (currently one
      cent per 10,000 SQS requests. It would take about 2.7 hours of polling
      for messages once per second to use up about 10,000 requests. At that
      rate, the cost per day would be about 9 cents).

      I'll write more about this, here and on my sites, as I make progress -
      and maybe post some code once it's readable enough.

      Erik Westermann
      http://blog.wWorkflow.net

      No virus found in this incoming message.
      Checked by AVG - www.avg.com
      Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date: 03/03/09 16:09:00


    • Erik Westermann
      Hi Udi, Amazon SQS makes some interesting integrations possible - one of the more obvious ones is having a publicly accessible, shared queue that allows
      Message 2 of 8 , Mar 15, 2009
      View Source
      • 0 Attachment
        Hi Udi,

        Amazon SQS makes some interesting integrations possible - one of the more obvious ones is having a publicly accessible, shared queue that allows integration between partners that don't want to have a lot of operations on the internet or in the cloud. This scenario allows two or more partners to share a third-party hosted and managed queue or storage location without requiring any of partners to have any operations on the internet.

        I thought about S3 and SimpleDb before going with SQS - I just felt SQS was closer to the existing MSMQ model.

        Based on my work so far, I am leaning towards having nServiceBus continue to work with MSMQ while using services like Amazon SQS (or whatever) under nServiceBus. I am leaning this way because the overall design of nServiceBus assumes connected, reliable, and fast communications are available, making it hard to convert the design into one that's capable of working in a less reliable, possibly slower environment. This appraoch is better in terms of longer term support since, in theory, it allows nServiceBus to evolve without having to make extensive changes to the existing code-base whenever somone wants to use something other than MSMQ.

        I'm not suggesting that it's not possible to decouple nServiceBus from MSMQ - I just think the approach of adding a layer under nServiceBus is easier to manage on an on-going basis.

        Regards,

        Erik
        http://ArtOfBabel.com


         

        --- In nservicebus@yahoogroups.com, "Udi Dahan" <thesoftwaresimplist@...> wrote:
        >
        > Erik,
        >
        >
        >
        > While it does make sense to use MSMQ as a subscription storage
        > implementation, Amazon SQS much less so. S3 would be a better choice.
        >
        >
        >
        > Hope that helps,
        >
        >
        >
        > -- Udi
        >
        >
        >
        >
        >
        > From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On
        > Behalf Of Erik Westermann
        > Sent: Thursday, March 05, 2009 1:21 AM
        > To: nservicebus@yahoogroups.com
        > Subject: [nservicebus] Working on Amazon SQS transport
        >
        >
        >
        > I'm working on adding to Amazon's SQS as a new transport in nServiceBus.
        > Amazon SQS (Simple Queue Service - information here:
        > http://aws.amazon.com/sqs/).
        >
        > This discusses some of the issues I ran into and how I addressed them.
        > You might be interested in this if you are considering adding your own
        > transport to nServiceBus, or are just curious about some of nServiceBus'
        > implementation techniques.
        >
        > My motivation for adding a remote, distributed queue to nServiceBus was
        > to gain a better understanding of nServiceBus and Amazon SQS. Using a
        > distributed, third party transport also offers some novel deployment
        > scenarios and capabilities.
        >
        > Amazon SQS is relatively easy to use and, fortunately, Amazon makes a
        > sample .NET library available to make it easier to work with. The
        > samples demonstrate how to create, list, and delete queues and how to
        > send and receive messages. Oddly enough, the sample code is not in line
        > with certain recommendations for working with SQS queues but once I got
        > around that, the services worked as documented.
        >
        > nServiceBus assumes connected, low latency communications with the
        > underlying storage - MSMQ. For example, when a publisher initializes, it
        > checks the subscriptions queue by iterating over all subscription
        > messages in the queue. The publisher does this to associate message
        > types with subscription queues, making it possible to route messages to
        > the right queue. The iteration occurs in a foreach loop based on a call
        > to MSMQ's GetAllMessages method.
        >
        > Transitioning the default connected model to Amazon SQS was not easy
        > since the disconnected nature of SQS needs to be hidden behind a
        > programming model that not only simulates a connected environment, but
        > also handles some of the SQS's features and capabilities. One of the
        > more significant issues is getting the total number of messages in an
        > SQS queue - Amazon SQS returns an approximate number of messages since
        > the underlying queue is highly distributed. Amazon warns against
        > fetching a specific number of messages, recomending instead to keep
        > polling for messages until you have a certian number of consecutive
        > null/empty receives. I did not have the patience to implement an
        > elaborate solution to address these issues, so I used a more basic
        > approach that's more likely to result in delayed messages (good enough
        > for now).
        >
        > Another aspect of the default nServiceBus transport is its tight
        > coupling with MSMQ - the coupling and other details are well hidden
        > behind the ITransport interface limiting it's impact to code within the
        > transport itself.
        >
        > An example of the tight coupling is when the nServiceBus MSMQ transport
        > uses an MSMQ message's Label and Id properties to handle initialization
        > and to determine a message's destination. This approach is efficient and
        > probably very fast.
        >
        > Personally, I would have created a new type of message and added a bit
        > more processing in my code rather than relying on features of the
        > underlying transport. Although my approach might end up being a little
        > slower and use more code it would work to not only make it easier to add
        > new transports, but also provide a greater degree of control over the
        > message itself. Using transport-specific attributes is analogous to
        > adding context to a message and representing that context using the
        > transport's capabilities. nSerivceBus' approach works well, I am just
        > mentioning it for others that might be interested in adding their own
        > transports.
        >
        > Configuration in nServiceBus is tricky - fortunately the included MSMQ
        > transport is easy to use as a model since it's easy to follow.
        >
        > Once things were working, I managed to send and receive messages using
        > the PubSub sample; however, the Server did not last long - it crashed
        > pretty quickly. My Server is crashing because my replacement for the
        > MSMQ Id and Label properties (and some other MSMQ-related code) is still
        > under development. The Subscriber, after a few tries, ran perfectly.
        >
        > Overall, my experience with nServiceBus 1.9 is positive and I am looking
        > forward to updates to the container model that's been mentioned here on
        > the list during the past few days. Amazon SQS is very robust and offers
        > a lot of features for the tiny price that they charge (currently one
        > cent per 10,000 SQS requests. It would take about 2.7 hours of polling
        > for messages once per second to use up about 10,000 requests. At that
        > rate, the cost per day would be about 9 cents).
        >
        > I'll write more about this, here and on my sites, as I make progress -
        > and maybe post some code once it's readable enough.
        >
        > Erik Westermann
        > http://blog.wWorkflow.net
        >
        >
        >
        > No virus found in this incoming message.
        > Checked by AVG - www.avg.com
        > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date: 03/03/09
        > 16:09:00
        >
      • Erik Westermann
        I agree, Ayende - there are a lot of assumptions throughout. I m thinking of adding a layer under the core nServiceBus (so to speak) to make it easier to add
        Message 3 of 8 , Mar 15, 2009
        View Source
        • 0 Attachment
          I agree, Ayende - there are a lot of assumptions throughout.

          I'm thinking of adding a layer under the core nServiceBus (so to speak)
          to make it easier to add support for alternate transports like Amazon
          SQS, etc.

          Regards,

          Erik
          http://ArtOfBabel.com


          --- In nservicebus@yahoogroups.com, Ayende Rahien <Ayende@...> wrote:
          >
          > Another issue is that a _lot_ of the assumptions that NSB makes about
          things
          > like rolling back a message, or whatever the queue can throw errors or
          not
          >
          > On Sat, Mar 14, 2009 at 7:35 AM, Udi Dahan
          thesoftwaresimplist@...wrote:
          >
          > > Erik,
          > >
          > >
          > >
          > > While it does make sense to use MSMQ as a subscription storage
          > > implementation, Amazon SQS much less so. S3 would be a better
          choice.
          > >
          > >
          > >
          > > Hope that helps,
          > >
          > >
          > >
          > > -- Udi
          > >
          > >
          > >
          > >
          > >
          > > *From:* nservicebus@yahoogroups.com
          [mailto:nservicebus@yahoogroups.com] *On
          > > Behalf Of *Erik Westermann
          > > *Sent:* Thursday, March 05, 2009 1:21 AM
          > > *To:* nservicebus@yahoogroups.com
          > > *Subject:* [nservicebus] Working on Amazon SQS transport
          > >
          > >
          > >
          > > I'm working on adding to Amazon's SQS as a new transport in
          nServiceBus.
          > > Amazon SQS (Simple Queue Service - information here:
          > > http://aws.amazon.com/sqs/).
          > >
          > > This discusses some of the issues I ran into and how I addressed
          them.
          > > You might be interested in this if you are considering adding your
          own
          > > transport to nServiceBus, or are just curious about some of
          nServiceBus'
          > > implementation techniques.
          > >
          > > My motivation for adding a remote, distributed queue to nServiceBus
          was
          > > to gain a better understanding of nServiceBus and Amazon SQS. Using
          a
          > > distributed, third party transport also offers some novel deployment
          > > scenarios and capabilities.
          > >
          > > Amazon SQS is relatively easy to use and, fortunately, Amazon makes
          a
          > > sample .NET library available to make it easier to work with. The
          > > samples demonstrate how to create, list, and delete queues and how
          to
          > > send and receive messages. Oddly enough, the sample code is not in
          line
          > > with certain recommendations for working with SQS queues but once I
          got
          > > around that, the services worked as documented.
          > >
          > > nServiceBus assumes connected, low latency communications with the
          > > underlying storage - MSMQ. For example, when a publisher
          initializes, it
          > > checks the subscriptions queue by iterating over all subscription
          > > messages in the queue. The publisher does this to associate message
          > > types with subscription queues, making it possible to route messages
          to
          > > the right queue. The iteration occurs in a foreach loop based on a
          call
          > > to MSMQ's GetAllMessages method.
          > >
          > > Transitioning the default connected model to Amazon SQS was not easy
          > > since the disconnected nature of SQS needs to be hidden behind a
          > > programming model that not only simulates a connected environment,
          but
          > > also handles some of the SQS's features and capabilities. One of the
          > > more significant issues is getting the total number of messages in
          an
          > > SQS queue - Amazon SQS returns an approximate number of messages
          since
          > > the underlying queue is highly distributed. Amazon warns against
          > > fetching a specific number of messages, recomending instead to keep
          > > polling for messages until you have a certian number of consecutive
          > > null/empty receives. I did not have the patience to implement an
          > > elaborate solution to address these issues, so I used a more basic
          > > approach that's more likely to result in delayed messages (good
          enough
          > > for now).
          > >
          > > Another aspect of the default nServiceBus transport is its tight
          > > coupling with MSMQ - the coupling and other details are well hidden
          > > behind the ITransport interface limiting it's impact to code within
          the
          > > transport itself.
          > >
          > > An example of the tight coupling is when the nServiceBus MSMQ
          transport
          > > uses an MSMQ message's Label and Id properties to handle
          initialization
          > > and to determine a message's destination. This approach is efficient
          and
          > > probably very fast.
          > >
          > > Personally, I would have created a new type of message and added a
          bit
          > > more processing in my code rather than relying on features of the
          > > underlying transport. Although my approach might end up being a
          little
          > > slower and use more code it would work to not only make it easier to
          add
          > > new transports, but also provide a greater degree of control over
          the
          > > message itself. Using transport-specific attributes is analogous to
          > > adding context to a message and representing that context using the
          > > transport's capabilities. nSerivceBus' approach works well, I am
          just
          > > mentioning it for others that might be interested in adding their
          own
          > > transports.
          > >
          > > Configuration in nServiceBus is tricky - fortunately the included
          MSMQ
          > > transport is easy to use as a model since it's easy to follow.
          > >
          > > Once things were working, I managed to send and receive messages
          using
          > > the PubSub sample; however, the Server did not last long - it
          crashed
          > > pretty quickly. My Server is crashing because my replacement for the
          > > MSMQ Id and Label properties (and some other MSMQ-related code) is
          still
          > > under development. The Subscriber, after a few tries, ran perfectly.
          > >
          > > Overall, my experience with nServiceBus 1.9 is positive and I am
          looking
          > > forward to updates to the container model that's been mentioned here
          on
          > > the list during the past few days. Amazon SQS is very robust and
          offers
          > > a lot of features for the tiny price that they charge (currently one
          > > cent per 10,000 SQS requests. It would take about 2.7 hours of
          polling
          > > for messages once per second to use up about 10,000 requests. At
          that
          > > rate, the cost per day would be about 9 cents).
          > >
          > > I'll write more about this, here and on my sites, as I make progress
          -
          > > and maybe post some code once it's readable enough.
          > >
          > > Erik Westermann
          > > http://blog.wWorkflow.net
          > >
          > > No virus found in this incoming message.
          > > Checked by AVG - www.avg.com
          > > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date:
          03/03/09
          > > 16:09:00
          > >
          > >
          > >
          >
        • Udi Dahan
          ... was closer to the existing MSMQ model. I m sorry to say there is not existing MSMQ model , there are both MSMQ and DB implementations of the subscription
          Message 4 of 8 , Mar 15, 2009
          View Source
          • 0 Attachment

            > I thought about S3 and SimpleDb before going with SQS - I just felt SQS was closer to the existing MSMQ model.

             

            I'm sorry to say there is not "existing MSMQ model", there are both MSMQ and DB implementations of the subscription storage interface. That may have been the cause of some confusion.

             

            The next thing to understand about the design of SQS is that it is best used from an EC2 instance, not from one's own self-hosted process.

            This may have also contributed to the difficulty in getting it to work.

             

            Finally, if you wish to perform internet-based integration between partners, that is a different model entirely than what the internal nServiceBus communication model is about.

             

            The way to do that is to set up a simple technological integration where you have a process that takes messages off of SQS (or any other internet "service bus") and puts them into a local MSMQ. Outgoing messages are handled similarly, with a different SQS queue and MSMQ queue, where you receive off the MSMQ and push to the SQS.

             

            Is that any clearer?

             

            With thanks,

             

            -- Udi

             

             

            From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Erik Westermann
            Sent: Sunday, March 15, 2009 4:34 PM
            To: nservicebus@yahoogroups.com
            Subject: [nservicebus] Re: Working on Amazon SQS transport

             

            Hi Udi,

            Amazon SQS makes some interesting integrations possible - one of the more obvious ones is having a publicly accessible, shared queue that allows integration between partners that don't want to have a lot of operations on the internet or in the cloud. This scenario allows two or more partners to share a third-party hosted and managed queue or storage location without requiring any of partners to have any operations on the internet.

            I thought about S3 and SimpleDb before going with SQS - I just felt SQS was closer to the existing MSMQ model.

            Based on my work so far, I am leaning towards having nServiceBus continue to work with MSMQ while using services like Amazon SQS (or whatever) under nServiceBus. I am leaning this way because the overall design of nServiceBus assumes connected, reliable, and fast communications are available, making it hard to convert the design into one that's capable of working in a less reliable, possibly slower environment. This appraoch is better in terms of longer term support since, in theory, it allows nServiceBus to evolve without having to make extensive changes to the existing code-base whenever somone wants to use something other than MSMQ.

            I'm not suggesting that it's not possible to decouple nServiceBus from MSMQ - I just think the approach of adding a layer under nServiceBus is easier to manage on an on-going basis.

            Regards,

            Erik
            http://ArtOfBabel.com


             

            --- In nservicebus@yahoogroups.com, "Udi Dahan" <thesoftwaresimplist@...> wrote:

            >
            > Erik,
            >
            >
            >
            > While it does make sense to use MSMQ as a subscription storage
            > implementation, Amazon SQS much less so. S3 would be a better choice.
            >
            >
            >
            > Hope that helps,
            >
            >
            >
            > -- Udi
            >
            >
            >
            >
            >
            > From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On
            > Behalf Of Erik Westermann
            > Sent: Thursday, March 05, 2009 1:21 AM
            > To: nservicebus@yahoogroups.com
            > Subject: [nservicebus] Working on Amazon SQS transport
            >
            >
            >
            > I'm working on adding to Amazon's SQS as a new transport in nServiceBus.
            > Amazon SQS (Simple Queue Service - information here:
            > http://aws.amazon.com/sqs/).
            >
            > This discusses some of the issues I ran into and how I addressed them.
            > You might be interested in this if you are considering adding your own
            > transport to nServiceBus, or are just curious about some of nServiceBus'
            > implementation techniques.
            >
            > My motivation for adding a remote, distributed queue to nServiceBus was
            > to gain a better understanding of nServiceBus and Amazon SQS. Using a
            > distributed, third party transport also offers some novel deployment
            > scenarios and capabilities.
            >
            > Amazon SQS is relatively easy to use and, fortunately, Amazon makes a
            > sample .NET library available to make it easier to work with. The
            > samples demonstrate how to create, list, and delete queues and how to
            > send and receive messages. Oddly enough, the sample code is not in line
            > with certain recommendations for working with SQS queues but once I got
            > around that, the services worked as documented.
            >
            > nServiceBus assumes connected, low latency communications with the
            > underlying storage - MSMQ. For example, when a publisher initializes, it
            > checks the subscriptions queue by iterating over all subscription
            > messages in the queue. The publisher does this to associate message
            > types with subscription queues, making it possible to route messages to
            > the right queue. The iteration occurs in a foreach loop based on a call
            > to MSMQ's GetAllMessages method.
            >
            > Transitioning the default connected model to Amazon SQS was not easy
            > since the disconnected nature of SQS needs to be hidden behind a
            > programming model that not only simulates a connected environment, but
            > also handles some of the SQS's features and capabilities. One of the
            > more significant issues is getting the total number of messages in an
            > SQS queue - Amazon SQS returns an approximate number of messages since
            > the underlying queue is highly distributed. Amazon warns against
            > fetching a specific number of messages, recomending instead to keep
            > polling for messages until you have a certian number of consecutive
            > null/empty receives. I did not have the patience to implement an
            > elaborate solution to address these issues, so I used a more basic
            > approach that's more likely to result in delayed messages (good enough
            > for now).
            >
            > Another aspect of the default nServiceBus transport is its tight
            > coupling with MSMQ - the coupling and other details are well hidden
            > behind the ITransport interface limiting it's impact to code within the
            > transport itself.
            >
            > An example of the tight coupling is when the nServiceBus MSMQ transport
            > uses an MSMQ message's Label and Id properties to handle initialization
            > and to determine a message's destination. This approach is efficient and
            > probably very fast.
            >
            > Personally, I would have created a new type of message and added a bit
            > more processing in my code rather than relying on features of the
            > underlying transport. Although my approach might end up being a little
            > slower and use more code it would work to not only make it easier to add
            > new transports, but also provide a greater degree of control over the
            > message itself. Using transport-specific attributes is analogous to
            > adding context to a message and representing that context using the
            > transport's capabilities. nSerivceBus' approach works well, I am just
            > mentioning it for others that might be interested in adding their own
            > transports.
            >
            > Configuration in nServiceBus is tricky - fortunately the included MSMQ
            > transport is easy to use as a model since it's easy to follow.
            >
            > Once things were working, I managed to send and receive messages using
            > the PubSub sample; however, the Server did not last long - it crashed
            > pretty quickly. My Server is crashing because my replacement for the
            > MSMQ Id and Label properties (and some other MSMQ-related code) is still
            > under development. The Subscriber, after a few tries, ran perfectly.
            >
            > Overall, my experience with nServiceBus 1.9 is positive and I am looking
            > forward to updates to the container model that's been mentioned here on
            > the list during the past few days. Amazon SQS is very robust and offers
            > a lot of features for the tiny price that they charge (currently one
            > cent per 10,000 SQS requests. It would take about 2.7 hours of polling
            > for messages once per second to use up about 10,000 requests. At that
            > rate, the cost per day would be about 9 cents).
            >
            > I'll write more about this, here and on my sites, as I make progress -
            > and maybe post some code once it's readable enough.
            >
            > Erik Westermann
            > http://blog.wWorkflow.net
            >
            >
            >
            > No virus found in this incoming message.
            > Checked by AVG - www.avg.com
            > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date: 03/03/09
            > 16:09:00
            >

          • Erik Westermann
            ... where you ... Yes - that s exactly the approach I m using. I had no problems getting things working - SQS simply uses a distributed queue thereby making a
            Message 5 of 8 , Mar 15, 2009
            View Source
            • 0 Attachment
              > The way to do that is to set up a simple technological integration
              where you
              > have a process that takes messages off of SQS (or any other internet..

              Yes - that's exactly the approach I'm using.

              I had no problems getting things working - SQS simply uses a distributed
              queue thereby making a call like "GetAllMessages" to an SQS queue less
              usable than the same call to an MSMQ. SQS also does not support labels
              and other properties of the MSMQ Message type, which get used a lot by
              the MSMQ transport.

              What I meant by the existing MSMQ model was the subscription storage
              interface. I wanted to learn more about nServiceBus by trying to add
              another subscription store and felt Amazon SQS made some options
              available.

              Regards,

              Erik


              --- In nservicebus@yahoogroups.com, "Udi Dahan"
              <thesoftwaresimplist@...> wrote:
              >
              > > I thought about S3 and SimpleDb before going with SQS - I just felt
              SQS
              > was closer to the existing MSMQ model.
              >
              >
              >
              > I'm sorry to say there is not "existing MSMQ model", there are both
              MSMQ and
              > DB implementations of the subscription storage interface. That may
              have been
              > the cause of some confusion.
              >
              >
              >
              > The next thing to understand about the design of SQS is that it is
              best used
              > from an EC2 instance, not from one's own self-hosted process.
              >
              > This may have also contributed to the difficulty in getting it to
              work.
              >
              >
              >
              > Finally, if you wish to perform internet-based integration between
              partners,
              > that is a different model entirely than what the internal nServiceBus
              > communication model is about.
              >
              >
              >
              > The way to do that is to set up a simple technological integration
              where you
              > have a process that takes messages off of SQS (or any other internet
              > "service bus") and puts them into a local MSMQ. Outgoing messages are
              > handled similarly, with a different SQS queue and MSMQ queue, where
              you
              > receive off the MSMQ and push to the SQS.
              >
              >
              >
              > Is that any clearer?
              >
              >
              >
              > With thanks,
              >
              >
              >
              > -- Udi
              >
              >
              >
              >
              >
              > From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com]
              On
              > Behalf Of Erik Westermann
              > Sent: Sunday, March 15, 2009 4:34 PM
              > To: nservicebus@yahoogroups.com
              > Subject: [nservicebus] Re: Working on Amazon SQS transport
              >
              >
              >
              > Hi Udi,
              >
              > Amazon SQS makes some interesting integrations possible - one of the
              more
              > obvious ones is having a publicly accessible, shared queue that allows
              > integration between partners that don't want to have a lot of
              operations on
              > the internet or in the cloud. This scenario allows two or more
              partners to
              > share a third-party hosted and managed queue or storage location
              without
              > requiring any of partners to have any operations on the internet.
              >
              > I thought about S3 and SimpleDb before going with SQS - I just felt
              SQS was
              > closer to the existing MSMQ model.
              >
              > Based on my work so far, I am leaning towards having nServiceBus
              continue to
              > work with MSMQ while using services like Amazon SQS (or whatever)
              under
              > nServiceBus. I am leaning this way because the overall design of
              nServiceBus
              > assumes connected, reliable, and fast communications are available,
              making
              > it hard to convert the design into one that's capable of working in a
              less
              > reliable, possibly slower environment. This appraoch is better in
              terms of
              > longer term support since, in theory, it allows nServiceBus to evolve
              > without having to make extensive changes to the existing code-base
              whenever
              > somone wants to use something other than MSMQ.
              >
              > I'm not suggesting that it's not possible to decouple nServiceBus from
              MSMQ
              > - I just think the approach of adding a layer under nServiceBus is
              easier to
              > manage on an on-going basis.
              >
              > Regards,
              >
              > Erik
              > http://ArtOfBabel.com
              >
              >
              >
              >
              > --- In nservicebus@yahoogroups.com, "Udi Dahan" thesoftwaresimplist@
              > wrote:
              > >
              > > Erik,
              > >
              > >
              > >
              > > While it does make sense to use MSMQ as a subscription storage
              > > implementation, Amazon SQS much less so. S3 would be a better
              choice.
              > >
              > >
              > >
              > > Hope that helps,
              > >
              > >
              > >
              > > -- Udi
              > >
              > >
              > >
              > >
              > >
              > > From: nservicebus@yahoogroups.com
              [mailto:nservicebus@yahoogroups.com] On
              > > Behalf Of Erik Westermann
              > > Sent: Thursday, March 05, 2009 1:21 AM
              > > To: nservicebus@yahoogroups.com
              > > Subject: [nservicebus] Working on Amazon SQS transport
              > >
              > >
              > >
              > > I'm working on adding to Amazon's SQS as a new transport in
              nServiceBus.
              > > Amazon SQS (Simple Queue Service - information here:
              > > http://aws.amazon.com/sqs/).
              > >
              > > This discusses some of the issues I ran into and how I addressed
              them.
              > > You might be interested in this if you are considering adding your
              own
              > > transport to nServiceBus, or are just curious about some of
              nServiceBus'
              > > implementation techniques.
              > >
              > > My motivation for adding a remote, distributed queue to nServiceBus
              was
              > > to gain a better understanding of nServiceBus and Amazon SQS. Using
              a
              > > distributed, third party transport also offers some novel deployment
              > > scenarios and capabilities.
              > >
              > > Amazon SQS is relatively easy to use and, fortunately, Amazon makes
              a
              > > sample .NET library available to make it easier to work with. The
              > > samples demonstrate how to create, list, and delete queues and how
              to
              > > send and receive messages. Oddly enough, the sample code is not in
              line
              > > with certain recommendations for working with SQS queues but once I
              got
              > > around that, the services worked as documented.
              > >
              > > nServiceBus assumes connected, low latency communications with the
              > > underlying storage - MSMQ. For example, when a publisher
              initializes, it
              > > checks the subscriptions queue by iterating over all subscription
              > > messages in the queue. The publisher does this to associate message
              > > types with subscription queues, making it possible to route messages
              to
              > > the right queue. The iteration occurs in a foreach loop based on a
              call
              > > to MSMQ's GetAllMessages method.
              > >
              > > Transitioning the default connected model to Amazon SQS was not easy
              > > since the disconnected nature of SQS needs to be hidden behind a
              > > programming model that not only simulates a connected environment,
              but
              > > also handles some of the SQS's features and capabilities. One of the
              > > more significant issues is getting the total number of messages in
              an
              > > SQS queue - Amazon SQS returns an approximate number of messages
              since
              > > the underlying queue is highly distributed. Amazon warns against
              > > fetching a specific number of messages, recomending instead to keep
              > > polling for messages until you have a certian number of consecutive
              > > null/empty receives. I did not have the patience to implement an
              > > elaborate solution to address these issues, so I used a more basic
              > > approach that's more likely to result in delayed messages (good
              enough
              > > for now).
              > >
              > > Another aspect of the default nServiceBus transport is its tight
              > > coupling with MSMQ - the coupling and other details are well hidden
              > > behind the ITransport interface limiting it's impact to code within
              the
              > > transport itself.
              > >
              > > An example of the tight coupling is when the nServiceBus MSMQ
              transport
              > > uses an MSMQ message's Label and Id properties to handle
              initialization
              > > and to determine a message's destination. This approach is efficient
              and
              > > probably very fast.
              > >
              > > Personally, I would have created a new type of message and added a
              bit
              > > more processing in my code rather than relying on features of the
              > > underlying transport. Although my approach might end up being a
              little
              > > slower and use more code it would work to not only make it easier to
              add
              > > new transports, but also provide a greater degree of control over
              the
              > > message itself. Using transport-specific attributes is analogous to
              > > adding context to a message and representing that context using the
              > > transport's capabilities. nSerivceBus' approach works well, I am
              just
              > > mentioning it for others that might be interested in adding their
              own
              > > transports.
              > >
              > > Configuration in nServiceBus is tricky - fortunately the included
              MSMQ
              > > transport is easy to use as a model since it's easy to follow.
              > >
              > > Once things were working, I managed to send and receive messages
              using
              > > the PubSub sample; however, the Server did not last long - it
              crashed
              > > pretty quickly. My Server is crashing because my replacement for the
              > > MSMQ Id and Label properties (and some other MSMQ-related code) is
              still
              > > under development. The Subscriber, after a few tries, ran perfectly.
              > >
              > > Overall, my experience with nServiceBus 1.9 is positive and I am
              looking
              > > forward to updates to the container model that's been mentioned here
              on
              > > the list during the past few days. Amazon SQS is very robust and
              offers
              > > a lot of features for the tiny price that they charge (currently one
              > > cent per 10,000 SQS requests. It would take about 2.7 hours of
              polling
              > > for messages once per second to use up about 10,000 requests. At
              that
              > > rate, the cost per day would be about 9 cents).
              > >
              > > I'll write more about this, here and on my sites, as I make progress
              -
              > > and maybe post some code once it's readable enough.
              > >
              > > Erik Westermann
              > > http://blog.wWorkflow.net
              > >
              > >
              > >
              > > No virus found in this incoming message.
              > > Checked by AVG - www.avg.com
              > > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date:
              03/03/09
              > > 16:09:00
              > >
              >
            • Udi Dahan
              Let me see if I can rephrase what I said to be clearer. Unless you re planning on running your system in Amazon EC2, it doesn t make sense to implement
              Message 6 of 8 , Mar 15, 2009
              View Source
              • 0 Attachment

                Let me see if I can rephrase what I said to be clearer.

                 

                Unless you're planning on running your system in Amazon EC2, it doesn't make sense to implement ITransport for SQS.

                Even if you were going to run your system in EC2, I'd advise against implementing ISubscriptionStorage on top of SQS.

                 

                It is important to make sure that the intent behind the abstractions are aligned with the technology one chooses otherwise certain capabilities (like fault tolerance) may be compromised.

                 

                Hope that is less equivocal than before.

                 

                With thanks,

                -- Udi

                 

                From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com] On Behalf Of Erik Westermann
                Sent: Monday, March 16, 2009 4:13 AM
                To: nservicebus@yahoogroups.com
                Subject: [nservicebus] Re: Working on Amazon SQS transport

                 

                > The way to do that is to set up a simple

                technological integration
                where you
                > have a process that takes messages off of SQS (or any other internet..

                Yes - that's exactly the approach I'm using.

                I had no problems getting things working - SQS simply uses a distributed
                queue thereby making a call like "GetAllMessages" to an SQS queue less
                usable than the same call to an MSMQ. SQS also does not support labels
                and other properties of the MSMQ Message type, which get used a lot by
                the MSMQ transport.

                What I meant by the existing MSMQ model was the subscription storage
                interface. I wanted to learn more about nServiceBus by trying to add
                another subscription store and felt Amazon SQS made some options
                available.

                Regards,

                Erik

                --- In nservicebus@yahoogroups.com, "Udi Dahan"
                <thesoftwaresimplist@...> wrote:
                >
                > > I thought about S3 and SimpleDb before going with SQS - I just felt
                SQS
                > was closer to the existing MSMQ model.
                >
                >
                >
                > I'm sorry to say there is not "existing MSMQ model", there are
                both
                MSMQ and
                > DB implementations of the subscription storage interface. That may
                have been
                > the cause of some confusion.
                >
                >
                >
                > The next thing to understand about the design of SQS is that it is
                best used
                > from an EC2 instance, not from one's own self-hosted process.
                >
                > This may have also contributed to the difficulty in getting it to
                work.
                >
                >
                >
                > Finally, if you wish to perform internet-based integration between
                partners,
                > that is a different model entirely than what the internal nServiceBus
                > communication model is about.
                >
                >
                >
                > The way to do that is to set up a simple technological integration
                where you
                > have a process that takes messages off of SQS (or any other internet
                > "service bus") and puts them into a local MSMQ. Outgoing
                messages are
                > handled similarly, with a different SQS queue and MSMQ queue, where
                you
                > receive off the MSMQ and push to the SQS.
                >
                >
                >
                > Is that any clearer?
                >
                >
                >
                > With thanks,
                >
                >
                >
                > -- Udi
                >
                >
                >
                >
                >
                > From: nservicebus@yahoogroups.com
                [mailto:nservicebus@yahoogroups.com]
                On
                > Behalf Of Erik Westermann
                > Sent: Sunday, March 15, 2009 4:34 PM
                > To: nservicebus@yahoogroups.com
                > Subject: [nservicebus] Re: Working on Amazon SQS transport
                >
                >
                >
                > Hi Udi,
                >
                > Amazon SQS makes some interesting integrations possible - one of the
                more
                > obvious ones is having a publicly accessible, shared queue that allows
                > integration between partners that don't want to have a lot of
                operations on
                > the internet or in the cloud. This scenario allows two or more
                partners to
                > share a third-party hosted and managed queue or storage location
                without
                > requiring any of partners to have any operations on the internet.
                >
                > I thought about S3 and SimpleDb before going with SQS - I just felt
                SQS was
                > closer to the existing MSMQ model.
                >
                > Based on my work so far, I am leaning towards having nServiceBus
                continue to
                > work with MSMQ while using services like Amazon SQS (or whatever)
                under
                > nServiceBus. I am leaning this way because the overall design of
                nServiceBus
                > assumes connected, reliable, and fast communications are available,
                making
                > it hard to convert the design into one that's capable of working in a
                less
                > reliable, possibly slower environment. This appraoch is better in
                terms of
                > longer term support since, in theory, it allows nServiceBus to evolve
                > without having to make extensive changes to the existing code-base
                whenever
                > somone wants to use something other than MSMQ.
                >
                > I'm not suggesting that it's not possible to decouple nServiceBus from
                MSMQ
                > - I just think the approach of adding a layer under nServiceBus is
                easier to
                > manage on an on-going basis.
                >
                > Regards,
                >
                > Erik
                > http://ArtOfBabel.com
                >
                >
                >
                >
                > --- In nservicebus@yahoogroups.com,
                "Udi Dahan" thesoftwaresimplist@
                > wrote:
                > >
                > > Erik,
                > >
                > >
                > >
                > > While it does make sense to use MSMQ as a subscription storage
                > > implementation, Amazon SQS much less so. S3 would be a better
                choice.
                > >
                > >
                > >
                > > Hope that helps,
                > >
                > >
                > >
                > > -- Udi
                > >
                > >
                > >
                > >
                > >
                > > From: nservicebus@yahoogroups.com
                [mailto:nservicebus@yahoogroups.com] On
                > > Behalf Of Erik Westermann
                > > Sent: Thursday, March 05, 2009 1:21 AM
                > > To: nservicebus@yahoogroups.com
                > > Subject: [nservicebus] Working on Amazon SQS transport
                > >
                > >
                > >
                > > I'm working on adding to Amazon's SQS as a new transport in
                nServiceBus.
                > > Amazon SQS (Simple Queue Service - information here:
                > > http://aws.amazon.com/sqs/).
                > >
                > > This discusses some of the issues I ran into and how I addressed
                them.
                > > You might be interested in this if you are considering adding your
                own
                > > transport to nServiceBus, or are just curious about some of
                nServiceBus'
                > > implementation techniques.
                > >
                > > My motivation for adding a remote, distributed queue to nServiceBus
                was
                > > to gain a better understanding of nServiceBus and Amazon SQS. Using
                a
                > > distributed, third party transport also offers some novel deployment
                > > scenarios and capabilities.
                > >
                > > Amazon SQS is relatively easy to use and, fortunately, Amazon makes
                a
                > > sample .NET library available to make it easier to work with. The
                > > samples demonstrate how to create, list, and delete queues and how
                to
                > > send and receive messages. Oddly enough, the sample code is not in
                line
                > > with certain recommendations for working with SQS queues but once I
                got
                > > around that, the services worked as documented.
                > >
                > > nServiceBus assumes connected, low latency communications with the
                > > underlying storage - MSMQ. For example, when a publisher
                initializes, it
                > > checks the subscriptions queue by iterating over all subscription
                > > messages in the queue. The publisher does this to associate message
                > > types with subscription queues, making it possible to route messages
                to
                > > the right queue. The iteration occurs in a foreach loop based on a
                call
                > > to MSMQ's GetAllMessages method.
                > >
                > > Transitioning the default connected model to Amazon SQS was not easy
                > > since the disconnected nature of SQS needs to be hidden behind a
                > > programming model that not only simulates a connected environment,
                but
                > > also handles some of the SQS's features and capabilities. One of the
                > > more significant issues is getting the total number of messages in
                an
                > > SQS queue - Amazon SQS returns an approximate number of messages
                since
                > > the underlying queue is highly distributed. Amazon warns against
                > > fetching a specific number of messages, recomending instead to keep
                > > polling for messages until you have a certian number of consecutive
                > > null/empty receives. I did not have the patience to implement an
                > > elaborate solution to address these issues, so I used a more basic
                > > approach that's more likely to result in delayed messages (good
                enough
                > > for now).
                > >
                > > Another aspect of the default nServiceBus transport is its tight
                > > coupling with MSMQ - the coupling and other details are well hidden
                > > behind the ITransport interface limiting it's impact to code within
                the
                > > transport itself.
                > >
                > > An example of the tight coupling is when the nServiceBus MSMQ
                transport
                > > uses an MSMQ message's Label and Id properties to handle
                initialization
                > > and to determine a message's destination. This approach is efficient
                and
                > > probably very fast.
                > >
                > > Personally, I would have created a new type of message and added a
                bit
                > > more processing in my code rather than relying on features of the
                > > underlying transport. Although my approach might end up being a
                little
                > > slower and use more code it would work to not only make it easier to
                add
                > > new transports, but also provide a greater degree of control over
                the
                > > message itself. Using transport-specific attributes is analogous to
                > > adding context to a message and representing that context using the
                > > transport's capabilities. nSerivceBus' approach works well, I am
                just
                > > mentioning it for others that might be interested in adding their
                own
                > > transports.
                > >
                > > Configuration in nServiceBus is tricky - fortunately the included
                MSMQ
                > > transport is easy to use as a model since it's easy to follow.
                > >
                > > Once things were working, I managed to send and receive messages
                using
                > > the PubSub sample; however, the Server did not last long - it
                crashed
                > > pretty quickly. My Server is crashing because my replacement for the
                > > MSMQ Id and Label properties (and some other MSMQ-related code) is
                still
                > > under development. The Subscriber, after a few tries, ran perfectly.
                > >
                > > Overall, my experience with nServiceBus 1.9 is positive and I am
                looking
                > > forward to updates to the container model that's been mentioned here
                on
                > > the list during the past few days. Amazon SQS is very robust and
                offers
                > > a lot of features for the tiny price that they charge (currently one
                > > cent per 10,000 SQS requests. It would take about 2.7 hours of
                polling
                > > for messages once per second to use up about 10,000 requests. At
                that
                > > rate, the cost per day would be about 9 cents).
                > >
                > > I'll write more about this, here and on my sites, as I make progress
                -
                > > and maybe post some code once it's readable enough.
                > >
                > > Erik Westermann
                > > http://blog.wWorkflow.net
                > >
                > >
                > >
                > > No virus found in this incoming message.
                > > Checked by AVG - www.avg.com
                > > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date:
                03/03/09
                > > 16:09:00
                > >
                >

              • Erik Westermann
                I m clear, thanks! Erik ... doesn t make ... are ... (like ... On ... internet.. ... distributed ... , ... felt ...
                Message 7 of 8 , Mar 16, 2009
                View Source
                • 0 Attachment
                  I'm clear, thanks!

                  Erik


                  --- In nservicebus@yahoogroups.com, "Udi Dahan"
                  <thesoftwaresimplist@...> wrote:
                  >
                  > Let me see if I can rephrase what I said to be clearer.
                  >
                  >
                  >
                  > Unless you're planning on running your system in Amazon EC2, it
                  doesn't make
                  > sense to implement ITransport for SQS.
                  >
                  > Even if you were going to run your system in EC2, I'd advise against
                  > implementing ISubscriptionStorage on top of SQS.
                  >
                  >
                  >
                  > It is important to make sure that the intent behind the abstractions
                  are
                  > aligned with the technology one chooses otherwise certain capabilities
                  (like
                  > fault tolerance) may be compromised.
                  >
                  >
                  >
                  > Hope that is less equivocal than before.
                  >
                  >
                  >
                  > With thanks,
                  >
                  > -- Udi
                  >
                  >
                  >
                  > From: nservicebus@yahoogroups.com [mailto:nservicebus@yahoogroups.com]
                  On
                  > Behalf Of Erik Westermann
                  > Sent: Monday, March 16, 2009 4:13 AM
                  > To: nservicebus@yahoogroups.com
                  > Subject: [nservicebus] Re: Working on Amazon SQS transport
                  >
                  >
                  >
                  > > The way to do that is to set up a simple technological integration
                  > where you
                  > > have a process that takes messages off of SQS (or any other
                  internet..
                  >
                  > Yes - that's exactly the approach I'm using.
                  >
                  > I had no problems getting things working - SQS simply uses a
                  distributed
                  > queue thereby making a call like "GetAllMessages" to an SQS queue less
                  > usable than the same call to an MSMQ. SQS also does not support labels
                  > and other properties of the MSMQ Message type, which get used a lot by
                  > the MSMQ transport.
                  >
                  > What I meant by the existing MSMQ model was the subscription storage
                  > interface. I wanted to learn more about nServiceBus by trying to add
                  > another subscription store and felt Amazon SQS made some options
                  > available.
                  >
                  > Regards,
                  >
                  > Erik
                  >
                  > --- In nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com> ,
                  > "Udi Dahan"
                  > thesoftwaresimplist@ wrote:
                  > >
                  > > > I thought about S3 and SimpleDb before going with SQS - I just
                  felt
                  > SQS
                  > > was closer to the existing MSMQ model.
                  > >
                  > >
                  > >
                  > > I'm sorry to say there is not "existing MSMQ model", there are both
                  > MSMQ and
                  > > DB implementations of the subscription storage interface. That may
                  > have been
                  > > the cause of some confusion.
                  > >
                  > >
                  > >
                  > > The next thing to understand about the design of SQS is that it is
                  > best used
                  > > from an EC2 instance, not from one's own self-hosted process.
                  > >
                  > > This may have also contributed to the difficulty in getting it to
                  > work.
                  > >
                  > >
                  > >
                  > > Finally, if you wish to perform internet-based integration between
                  > partners,
                  > > that is a different model entirely than what the internal
                  nServiceBus
                  > > communication model is about.
                  > >
                  > >
                  > >
                  > > The way to do that is to set up a simple technological integration
                  > where you
                  > > have a process that takes messages off of SQS (or any other internet
                  > > "service bus") and puts them into a local MSMQ. Outgoing messages
                  are
                  > > handled similarly, with a different SQS queue and MSMQ queue, where
                  > you
                  > > receive off the MSMQ and push to the SQS.
                  > >
                  > >
                  > >
                  > > Is that any clearer?
                  > >
                  > >
                  > >
                  > > With thanks,
                  > >
                  > >
                  > >
                  > > -- Udi
                  > >
                  > >
                  > >
                  > >
                  > >
                  > > From: nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com>
                  > [mailto:nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com> ]
                  > On
                  > > Behalf Of Erik Westermann
                  > > Sent: Sunday, March 15, 2009 4:34 PM
                  > > To: nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com>
                  > > Subject: [nservicebus] Re: Working on Amazon SQS transport
                  > >
                  > >
                  > >
                  > > Hi Udi,
                  > >
                  > > Amazon SQS makes some interesting integrations possible - one of the
                  > more
                  > > obvious ones is having a publicly accessible, shared queue that
                  allows
                  > > integration between partners that don't want to have a lot of
                  > operations on
                  > > the internet or in the cloud. This scenario allows two or more
                  > partners to
                  > > share a third-party hosted and managed queue or storage location
                  > without
                  > > requiring any of partners to have any operations on the internet.
                  > >
                  > > I thought about S3 and SimpleDb before going with SQS - I just felt
                  > SQS was
                  > > closer to the existing MSMQ model.
                  > >
                  > > Based on my work so far, I am leaning towards having nServiceBus
                  > continue to
                  > > work with MSMQ while using services like Amazon SQS (or whatever)
                  > under
                  > > nServiceBus. I am leaning this way because the overall design of
                  > nServiceBus
                  > > assumes connected, reliable, and fast communications are available,
                  > making
                  > > it hard to convert the design into one that's capable of working in
                  a
                  > less
                  > > reliable, possibly slower environment. This appraoch is better in
                  > terms of
                  > > longer term support since, in theory, it allows nServiceBus to
                  evolve
                  > > without having to make extensive changes to the existing code-base
                  > whenever
                  > > somone wants to use something other than MSMQ.
                  > >
                  > > I'm not suggesting that it's not possible to decouple nServiceBus
                  from
                  > MSMQ
                  > > - I just think the approach of adding a layer under nServiceBus is
                  > easier to
                  > > manage on an on-going basis.
                  > >
                  > > Regards,
                  > >
                  > > Erik
                  > > http://ArtOfBabel.com
                  > >
                  > >
                  > >
                  > >
                  > > --- In nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com>
                  > , "Udi Dahan" thesoftwaresimplist@
                  > > wrote:
                  > > >
                  > > > Erik,
                  > > >
                  > > >
                  > > >
                  > > > While it does make sense to use MSMQ as a subscription storage
                  > > > implementation, Amazon SQS much less so. S3 would be a better
                  > choice.
                  > > >
                  > > >
                  > > >
                  > > > Hope that helps,
                  > > >
                  > > >
                  > > >
                  > > > -- Udi
                  > > >
                  > > >
                  > > >
                  > > >
                  > > >
                  > > > From: nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com>
                  >
                  > [mailto:nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com> ]
                  > On
                  > > > Behalf Of Erik Westermann
                  > > > Sent: Thursday, March 05, 2009 1:21 AM
                  > > > To: nservicebus@yahoogroups.com
                  <mailto:nservicebus%40yahoogroups.com>
                  > > > Subject: [nservicebus] Working on Amazon SQS transport
                  > > >
                  > > >
                  > > >
                  > > > I'm working on adding to Amazon's SQS as a new transport in
                  > nServiceBus.
                  > > > Amazon SQS (Simple Queue Service - information here:
                  > > > http://aws.amazon.com/sqs/).
                  > > >
                  > > > This discusses some of the issues I ran into and how I addressed
                  > them.
                  > > > You might be interested in this if you are considering adding your
                  > own
                  > > > transport to nServiceBus, or are just curious about some of
                  > nServiceBus'
                  > > > implementation techniques.
                  > > >
                  > > > My motivation for adding a remote, distributed queue to
                  nServiceBus
                  > was
                  > > > to gain a better understanding of nServiceBus and Amazon SQS.
                  Using
                  > a
                  > > > distributed, third party transport also offers some novel
                  deployment
                  > > > scenarios and capabilities.
                  > > >
                  > > > Amazon SQS is relatively easy to use and, fortunately, Amazon
                  makes
                  > a
                  > > > sample .NET library available to make it easier to work with. The
                  > > > samples demonstrate how to create, list, and delete queues and how
                  > to
                  > > > send and receive messages. Oddly enough, the sample code is not in
                  > line
                  > > > with certain recommendations for working with SQS queues but once
                  I
                  > got
                  > > > around that, the services worked as documented.
                  > > >
                  > > > nServiceBus assumes connected, low latency communications with the
                  > > > underlying storage - MSMQ. For example, when a publisher
                  > initializes, it
                  > > > checks the subscriptions queue by iterating over all subscription
                  > > > messages in the queue. The publisher does this to associate
                  message
                  > > > types with subscription queues, making it possible to route
                  messages
                  > to
                  > > > the right queue. The iteration occurs in a foreach loop based on a
                  > call
                  > > > to MSMQ's GetAllMessages method.
                  > > >
                  > > > Transitioning the default connected model to Amazon SQS was not
                  easy
                  > > > since the disconnected nature of SQS needs to be hidden behind a
                  > > > programming model that not only simulates a connected environment,
                  > but
                  > > > also handles some of the SQS's features and capabilities. One of
                  the
                  > > > more significant issues is getting the total number of messages in
                  > an
                  > > > SQS queue - Amazon SQS returns an approximate number of messages
                  > since
                  > > > the underlying queue is highly distributed. Amazon warns against
                  > > > fetching a specific number of messages, recomending instead to
                  keep
                  > > > polling for messages until you have a certian number of
                  consecutive
                  > > > null/empty receives. I did not have the patience to implement an
                  > > > elaborate solution to address these issues, so I used a more basic
                  > > > approach that's more likely to result in delayed messages (good
                  > enough
                  > > > for now).
                  > > >
                  > > > Another aspect of the default nServiceBus transport is its tight
                  > > > coupling with MSMQ - the coupling and other details are well
                  hidden
                  > > > behind the ITransport interface limiting it's impact to code
                  within
                  > the
                  > > > transport itself.
                  > > >
                  > > > An example of the tight coupling is when the nServiceBus MSMQ
                  > transport
                  > > > uses an MSMQ message's Label and Id properties to handle
                  > initialization
                  > > > and to determine a message's destination. This approach is
                  efficient
                  > and
                  > > > probably very fast.
                  > > >
                  > > > Personally, I would have created a new type of message and added a
                  > bit
                  > > > more processing in my code rather than relying on features of the
                  > > > underlying transport. Although my approach might end up being a
                  > little
                  > > > slower and use more code it would work to not only make it easier
                  to
                  > add
                  > > > new transports, but also provide a greater degree of control over
                  > the
                  > > > message itself. Using transport-specific attributes is analogous
                  to
                  > > > adding context to a message and representing that context using
                  the
                  > > > transport's capabilities. nSerivceBus' approach works well, I am
                  > just
                  > > > mentioning it for others that might be interested in adding their
                  > own
                  > > > transports.
                  > > >
                  > > > Configuration in nServiceBus is tricky - fortunately the included
                  > MSMQ
                  > > > transport is easy to use as a model since it's easy to follow.
                  > > >
                  > > > Once things were working, I managed to send and receive messages
                  > using
                  > > > the PubSub sample; however, the Server did not last long - it
                  > crashed
                  > > > pretty quickly. My Server is crashing because my replacement for
                  the
                  > > > MSMQ Id and Label properties (and some other MSMQ-related code) is
                  > still
                  > > > under development. The Subscriber, after a few tries, ran
                  perfectly.
                  > > >
                  > > > Overall, my experience with nServiceBus 1.9 is positive and I am
                  > looking
                  > > > forward to updates to the container model that's been mentioned
                  here
                  > on
                  > > > the list during the past few days. Amazon SQS is very robust and
                  > offers
                  > > > a lot of features for the tiny price that they charge (currently
                  one
                  > > > cent per 10,000 SQS requests. It would take about 2.7 hours of
                  > polling
                  > > > for messages once per second to use up about 10,000 requests. At
                  > that
                  > > > rate, the cost per day would be about 9 cents).
                  > > >
                  > > > I'll write more about this, here and on my sites, as I make
                  progress
                  > -
                  > > > and maybe post some code once it's readable enough.
                  > > >
                  > > > Erik Westermann
                  > > > http://blog.wWorkflow.net
                  > > >
                  > > >
                  > > >
                  > > > No virus found in this incoming message.
                  > > > Checked by AVG - www.avg.com
                  > > > Version: 8.0.237 / Virus Database: 270.11.3/1970 - Release Date:
                  > 03/03/09
                  > > > 16:09:00
                  > > >
                  > >
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.