Loading ...
Sorry, an error occurred while loading the content.

RE: [rest-discuss] REST & Internal Integration

Expand Messages
  • Mike Dierken
    Mike, Sorry I haven t responded sooner... To make up for it, I wrote a really long response. I don t have anything against JMS or messaging systems, they are
    Message 1 of 12 , Apr 29, 2006
    • 0 Attachment
      Mike,
      Sorry I haven't responded sooner... To make up for it, I wrote a really long
      response.

      I don't have anything against JMS or messaging systems, they are great
      tools. And just like HTTP, they work wonders when used in the right
      situation. This thread is interesting to me because there hasn't been much
      discussion about applying aspects of REST to a classical asynchronous
      messaging based system.
      In my opinion, there are a couple things that are similar between the two -
      simple and common interactions for many kinds of applications (send message,
      receive message, etc), payload formats independent of message framing, and
      so on. But there are a few things common in REST that I don't see in JMS or
      other messaging systems :

      1) application semantics : sending a message in REST means something
      (delete, modify, retrieve, etc), whereas sending a message in JMS or other
      messaging systems is just sending a message (and it usually has unknown
      consequences, which requires endless coordination to work out the details)

      2) resource identifiers : this sort of follows from being an application
      protocol - the protocol is all about interacting with resources, which
      provides a framework for modelling applications (above just moving bytes
      from point a to point b).

      3) hypertext (as the engine of application state) : this is one of the
      facets of REST that took me the longest to put into practice, but has paid
      off very well. Using documents with tags that help pick out the resource
      identifiers is invaluable in a dynamic, adaptive and configuration-free
      system. Many systems have dynamically retrieved configurations, and this
      provides part of the benefit of hypertext, and when links (resource
      identifiers that can be interacted with) are used a lot of config file
      processing becomes generalizable and just falls away. For example, at work
      we recently built a large scale real-time data publishing system and the
      consuming systems have a link to a 'directory service', which is an xml blob
      with elements with links to the subjects to subscribe to, links to the
      cold-start data to bootstrap with and so on. The cold-start data is somewhat
      large, but by teaching the clients to 'just follow the links', the data
      could be arbitrarily partitioned by the server. In addition, with a little
      thought you could put multiple links in there and have parallel streams
      being retrieved by the client - a good way to utilize all that bandwidth
      just laying around.


      > I spend most of my time working on a system that has approximately 50
      > services. The services are pretty stupid, all they know is
      > they get an event XML input. They typically respond with an
      > XML event. They listen on a set of JMS Topics or a Topic
      > pattern (e.g., COMPANYNAME.LINEOFBUSINESS.NOUN.*).
      > They also typically respond to set of Topics. What I like
      > about JMS in the system is the guaranteed delivery & durable
      > subscriptions. The service impls don't care about the network
      > because they know that JMS is there.
      These are extremely useful tools - durable subscriptions, guaranteed
      delivery, etc. But from my point of view the 'service' isn't all that
      'stupid'. The service is everything the outside world would see, and that
      includes those handy things like disconnected use and reliability. The code
      that plugs into this is simplified by that environment, which is hufely
      valuable. I'm not a messaging or protocol expert, but I always thought that
      HTTP over TCP/IP provided guaranteed delivery within the connection - I'd
      like to hear from others on how we should consider the 'guarantee-ness' of
      HTTP (within a connection). The extra complexity of guaranteed delivery
      outside of a TCP/IP connection provides support for the disconnected use
      cases you mention, and I think this would be a great thing for a Web-scale
      system to provide, but I don't see it happening yet and I'm not sure why.


      > Also, durable subscriptions are great [...]
      > I would really like to see an HTTP based pub/sub like this.
      > Idon't know enough about REST to know if it would be evil to do.
      > I babbled about that a bit more here:
      > http://fuzzypanic.blogspot.com/2006/03/httppox-xml-based-eda-pub-sub.html
      > Let me know if you know of anything.
      There have been a few companies doing that - you found mod_pubsub already,
      which was a spin-out from KnowNow (which I helped start up many years ago,
      for a short period of time). Supposedly Renkoo may be doing work to make
      mod-pubsub live again. The toy 'messaging' web app I linked to last time
      shows that it's very easy to build messaging with pure HTTP, and with modern
      systems supporting non-blocking IO, it's not too hard to be efficient. The
      hard part is the pump that keeps the messages flowing along the durable
      subscriptions, dealing with cycles and other plumbing issues.

      You ask a very good question though - is pubsub compatible with REST? If
      REST is about application level semantics, you could model messages,
      subscriptions, etc. as resources and go that route. But then what are the
      messages about? How are those raised to the application level so that common
      semantics can be applied?


      > Sure, but that block means something typically - it means the
      > message was persisted so the client doesn't have to ferret it
      > away just in case something bad happens later.
      > With Web Style, you can do that too - my point is you have to do that
      > yourself (client ferret away the data that was PUT/POST).
      I see this as partly a difference in whether the client host or the server
      host has some persistent storage. If there is no client persistent storage,
      then the client application logic needs to know when the server has taken
      hold of the data in some persistent fashion (if that's the goal in that
      situation). The other part is how well the application behaves in the face
      of an outage or disconnection - picking up where you left off is harder if
      the application doesn't store the state as it makes progress. My hunch is
      that making progress when outbound messages are not being delivered isn't a
      huge win most of the time - if there was work that could be accomplished
      without outbound network traffic, then it should be accomplished regardless
      of the messaging style.


      > Nope, I mean make pub/sub more RESTful.
      > Like I said, I don't know enough about REST principals to know if
      > this is evil or not. I have a decent messaging middleware product today.
      > I'm happy with it - I just want to know if its overkill or not or
      > if there is a simpler way. If some of my integration types
      > are best handled via JMS, fine with me. I just want to ask
      > the "stupid question" - to make sure I'm not thinking JMS is
      > better then it is.
      JMS is fine. I have to ask though, how often do you use JMS to integrate
      with groups outside your control, or outside your company?

      > > Here is a sample app I wrote a while back that does what you are talking
      about :
      > >
      http://www.searchalert.net:8080/dierken/events/chat.html?topic=rest-discuss
      > > The underlying system holds messages here:
      http://www.searchalert.net:8080/xmlrouter/msgs/rest-discuss
      > So what is on the server - is there messaging behind it?
      Nope, just a 50 line servlet and the messages in-memory in a linked list.



      > > Are you sure you didn't mean N+1 responses?
      > I'm talking about a client sending 1 request, having N
      > services subscribe and send a response to the client.
      I was kidding. The issue I was alluding to is - how do you know you've
      received all the messages or if there is still one more straggler? The
      pubsub approach is 'supposed' to be about loosely coupled systems, but if
      the publisher has to change to meet the specifications of each consumer,
      then that loose coupling has just tightened.
    • Claude Montpetit
      ... I ve been playing with both JMS and REST recently and at first, I was trying to see how REST could be used instead of JMS. I came to think that these 2
      Message 2 of 12 , May 1, 2006
      • 0 Attachment
        Mike Dierken a écrit :
        > Mike,
        > Sorry I haven't responded sooner... To make up for it, I wrote a really long
        > response.
        >
        > I don't have anything against JMS or messaging systems, they are great
        > tools. And just like HTTP, they work wonders when used in the right
        > situation. This thread is interesting to me because there hasn't been much
        > discussion about applying aspects of REST to a classical asynchronous
        > messaging based system.

        I've been playing with both JMS and REST recently and at first, I was
        trying to see how REST could be used instead of JMS. I came to think
        that these 2 "specifications" should not be seen as one-or-the-other.
        They are 2 different technologies that solve 2 different sets of
        problem; they can play together nicely when facilitating access to a JMS
        service is a requirement.

        In our system, we have a bunch JMS services (queue receivers and topic
        consumers). We provide custom Java client classes that clients can use
        to easily connect to the services. These talk directly to the JMS
        brokers to contact our JMS services.

        There is (well, will be) also a REST interface to the JMS services. The
        REST service allows the creation of a service request through a simple
        POST to a URI. The implementation of this REST "service" connects to the
        JMS broker, sends the corresponding message, and waits (with timeout)
        until it receives confirmation that the message was received by the JMS
        service. This confirmation of reception implies that the service request
        has created a resource and this new resource is returned to the REST
        client who can, at anytime, GET it to see the status of his orignal
        request (running, completed, cancelled, logging ...).

        So far, the JMS custom client classes are only used internally, and
        while the system has not yet been used by our clients, I expect they
        will prefer to use the REST access to the service. Even though the Java
        JMS custom classes provide finer control, REST' simplicity will most
        likely be a real advantage.

        Technically, from REST's client perspective, the launching of the
        service is synchronous: the REST server connects to the JMS service,
        waits for confirmation of reception, and returns the resource URI to the
        client. But the execution of the service request is asynchronous, and
        the REST clients can make calls on the returned URI to check the status
        of the JMS service request.



        > [...]
      • Mike Herrick
        Hi Mike, Thanks for the response. I wrote you back a missive myself ;) I hope someone other then the two of us finds this conversation useful. I d love to see
        Message 3 of 12 , May 1, 2006
        • 0 Attachment
          Hi Mike,

          Thanks for the response. I wrote you back a missive myself ;) I hope
          someone other then the two of us finds this conversation useful. I'd
          love to see a line drawn in the sand on whether what we are talking
          about is achievable / appropriate for REST or not. The WS-* world is
          going after stuff like this. Maybe RESTafarians will say it isn't
          appropriate, maybe it is and some impls will happen.

          You hit it on the head on why REST appeals to me when you said, "...
          payload formats independent of message framing". I look at SOAP and
          especially some of the more recent WS-* specs and see all that cruft
          right in the payload - just is awful IMHO. At first glance, the fact
          that SOAP does this seems largely immaterial ... but once you dig
          deeper you see what a negative impact it has.

          I was also thinking a few weeks ago when I read up some more on REST
          on the similarity that JMS does share with SOAP - that there are no
          application semantics as you called it. In JMS/messaging, its just a
          message. The verb is somewhere in the Destination name or somewhere
          in the XML (we hope). This is tolerable IMHO in JMS because it is a
          defined integration platform where SOAP is going over HTTP, which, is
          used for web sites and integration. I read something on SOAP vs. REST
          security & a movement a while back on forcing SOAP/HTTP to go over
          another port then 80 ... my point is at least JMS has this going for
          it. I don't know either way if I think its appropriate for JMS to
          have verbs (i.e., GET, PUT, POST, DELETE, etc.). But you are
          definitely correct that the fact that JMS does not have application
          semantics results in coordination etc. I'm very careful about this in
          my designs, but lots of people aren't.

          Sounds pretty interesting on the config - I'm cool with all config
          mechanisms that aren't JNDI (sorry I spent too many years in J2EE).
          What you describe is actually how our JMS provider works. It needs
          the config at broker start time, but caches the last good copy in
          case the management broker gets horked.

          On "stupid" services ... they are only "stupid" because they
          implement an interface that deals with all the JMS stuff. So the
          impls. really are trivial. But you are right that at the end of the
          day, developers do know a lot about JMS - they have to. My point was
          more around the details a developer doesn't have to deal with in
          terms of manually acking delivery, saving things in a temporary data
          store, etc. I'm curios on the HTTP & guaranteed delivery w/i the
          session - doesn't sound like the same thing though. With "persistent"
          messaging in JMS, the provider must write the message to disk before
          returning. This means that (unless you clown up the receiver by
          acking the message when you shouldn't), your message is going to get
          somewhere and not get lost. And if you don't ack it, it will be
          sitting on the Queue/Topic and will be resent to the receiver etc.
          This is very useful because your services don't have to account for
          this. A lot of the people I talk to who just use WS-* look at me like
          I'm really hard core for worrying about this - I think they are a
          little incompetent for not ;) When JMS is used correctly, you get
          very clear success/failures - messages either get processed correctly
          or they get sent to an error queue. You know what is going on. Not
          saying you don't w/o it, but beats the heck out of greping through
          log files. And to be fair, I've seen people screw this up with JMS so
          you don't just get it for free.

          You said, "The extra complexity of guaranteed delivery outside of a
          TCP/IP connection provides support for the disconnected use cases you
          mention, and I think this would be a great thing for a Web-scale
          system to provide, but I don't see it happening yet and I'm not sure
          why." Good to hear - so do I.

          Next you said, "My hunch is that making progress when outbound
          messages are not being delivered isn't a huge win most of the time -
          if there was work that could be accomplished without outbound network
          traffic, then it should be accomplished regardless of the messaging
          style." In my world, it is a huge win. It is a huge win when you have
          a logical transaction (e.g., submit insurance) that spans several
          services. That message could make it through 3 of the 8 services
          involved. It might make it through validation, rules engine, tiering,
          and rating, but then fail on the way to getting imaged & sent to some
          system of record. The original client (the web app where a mouse
          click happened) is long gone at this point. Lets say that for some
          reason the service that was supposed to image the insurance
          application couldn't talk to the imaging service because a router
          failed. No problem with proper JMS/messaging ... that message either
          gets routed to an error queue or "rolled back / paused" to messaging
          for future redelivery at the pause interval. Its the cumulative
          complexity of service interaction that results in messaging being
          beneficial - if you have just a couple services, you can do without
          it - its when there is coordination between the services where things
          get hairy.

          Next you said, "JMS is fine. I have to ask though, how often do you
          use JMS to integrate with groups outside your control, or outside
          your company?"

          Bingo - this is where it falls down. My company has several JMS
          providers. When certain organizations need to communicate, they need
          to bridge the providers. Not ideal obviously. This is the type of
          thing that makes me wonder if there is a better way. WS-
          ReliableMessaging is supposed to do this, but I'm not a fan of WS-
          anything. I'd love to see a reliable REST and a pub/sub - event
          driven REST if it can be done w/o hurting the web etc.

          We aren't disagreeing with each other, but you are proving my point
          when you say, "Nope, just a 50 line servlet and the messages in-
          memory in a linked list." Now, you are smart and got this right I'm
          sure - I am positive my JMS vendor did. I just freak out about
          various developers having to write their own impl. Sure you could
          have a little in house framework for this - not hard, but it is
          harder then we think to mandate etc.

          Lastly, you said: "I was kidding. The issue I was alluding to is -
          how do you know you've received all the messages or if there is still
          one more straggler? The pubsub approach is 'supposed' to be about
          loosely coupled systems, but if the publisher has to change to meet
          the specifications of each consumer, then that loose coupling has
          just tightened." Again, great point. People often do this because
          they know who is listening - they wait for the N responses (they know
          the value of N) and the return the Set of responses. Even though you
          slow things down, the only way to do this properly is to mandate a
          constant timeout (i.e., the request/reply pub/sub always takes 5
          seconds) regardless if every service who can respond and did respond
          did so in 3 seconds.

          Thanks again for the comments.

          Mike



          On Apr 29, 2006, at 11:29 PM, Mike Dierken wrote:

          > Mike,
          > Sorry I haven't responded sooner... To make up for it, I wrote a
          > really long
          > response.
          >
          > I don't have anything against JMS or messaging systems, they are great
          > tools. And just like HTTP, they work wonders when used in the right
          > situation. This thread is interesting to me because there hasn't
          > been much
          > discussion about applying aspects of REST to a classical asynchronous
          > messaging based system.
          > In my opinion, there are a couple things that are similar between
          > the two -
          > simple and common interactions for many kinds of applications (send
          > message,
          > receive message, etc), payload formats independent of message
          > framing, and
          > so on. But there are a few things common in REST that I don't see
          > in JMS or
          > other messaging systems :
          >
          > 1) application semantics : sending a message in REST means something
          > (delete, modify, retrieve, etc), whereas sending a message in JMS
          > or other
          > messaging systems is just sending a message (and it usually has
          > unknown
          > consequences, which requires endless coordination to work out the
          > details)
          >
          > 2) resource identifiers : this sort of follows from being an
          > application
          > protocol - the protocol is all about interacting with resources, which
          > provides a framework for modelling applications (above just moving
          > bytes
          > from point a to point b).
          >
          > 3) hypertext (as the engine of application state) : this is one of the
          > facets of REST that took me the longest to put into practice, but
          > has paid
          > off very well. Using documents with tags that help pick out the
          > resource
          > identifiers is invaluable in a dynamic, adaptive and configuration-
          > free
          > system. Many systems have dynamically retrieved configurations, and
          > this
          > provides part of the benefit of hypertext, and when links (resource
          > identifiers that can be interacted with) are used a lot of config file
          > processing becomes generalizable and just falls away. For example,
          > at work
          > we recently built a large scale real-time data publishing system
          > and the
          > consuming systems have a link to a 'directory service', which is an
          > xml blob
          > with elements with links to the subjects to subscribe to, links to the
          > cold-start data to bootstrap with and so on. The cold-start data is
          > somewhat
          > large, but by teaching the clients to 'just follow the links', the
          > data
          > could be arbitrarily partitioned by the server. In addition, with a
          > little
          > thought you could put multiple links in there and have parallel
          > streams
          > being retrieved by the client - a good way to utilize all that
          > bandwidth
          > just laying around.
          >
          >
          >> I spend most of my time working on a system that has approximately 50
          >> services. The services are pretty stupid, all they know is
          >> they get an event XML input. They typically respond with an
          >> XML event. They listen on a set of JMS Topics or a Topic
          >> pattern (e.g., COMPANYNAME.LINEOFBUSINESS.NOUN.*).
          >> They also typically respond to set of Topics. What I like
          >> about JMS in the system is the guaranteed delivery & durable
          >> subscriptions. The service impls don't care about the network
          >> because they know that JMS is there.
          > These are extremely useful tools - durable subscriptions, guaranteed
          > delivery, etc. But from my point of view the 'service' isn't all that
          > 'stupid'. The service is everything the outside world would see,
          > and that
          > includes those handy things like disconnected use and reliability.
          > The code
          > that plugs into this is simplified by that environment, which is
          > hufely
          > valuable. I'm not a messaging or protocol expert, but I always
          > thought that
          > HTTP over TCP/IP provided guaranteed delivery within the connection
          > - I'd
          > like to hear from others on how we should consider the 'guarantee-
          > ness' of
          > HTTP (within a connection). The extra complexity of guaranteed
          > delivery
          > outside of a TCP/IP connection provides support for the
          > disconnected use
          > cases you mention, and I think this would be a great thing for a
          > Web-scale
          > system to provide, but I don't see it happening yet and I'm not
          > sure why.
          >
          >
          >> Also, durable subscriptions are great [...]
          >> I would really like to see an HTTP based pub/sub like this.
          >> Idon't know enough about REST to know if it would be evil to do.
          >> I babbled about that a bit more here:
          >> http://fuzzypanic.blogspot.com/2006/03/httppox-xml-based-eda-pub-
          >> sub.html
          >> Let me know if you know of anything.
          > There have been a few companies doing that - you found mod_pubsub
          > already,
          > which was a spin-out from KnowNow (which I helped start up many
          > years ago,
          > for a short period of time). Supposedly Renkoo may be doing work to
          > make
          > mod-pubsub live again. The toy 'messaging' web app I linked to last
          > time
          > shows that it's very easy to build messaging with pure HTTP, and
          > with modern
          > systems supporting non-blocking IO, it's not too hard to be
          > efficient. The
          > hard part is the pump that keeps the messages flowing along the
          > durable
          > subscriptions, dealing with cycles and other plumbing issues.
          >
          > You ask a very good question though - is pubsub compatible with
          > REST? If
          > REST is about application level semantics, you could model messages,
          > subscriptions, etc. as resources and go that route. But then what
          > are the
          > messages about? How are those raised to the application level so
          > that common
          > semantics can be applied?
          >
          >
          >> Sure, but that block means something typically - it means the
          >> message was persisted so the client doesn't have to ferret it
          >> away just in case something bad happens later.
          >> With Web Style, you can do that too - my point is you have to do that
          >> yourself (client ferret away the data that was PUT/POST).
          > I see this as partly a difference in whether the client host or the
          > server
          > host has some persistent storage. If there is no client persistent
          > storage,
          > then the client application logic needs to know when the server has
          > taken
          > hold of the data in some persistent fashion (if that's the goal in
          > that
          > situation). The other part is how well the application behaves in
          > the face
          > of an outage or disconnection - picking up where you left off is
          > harder if
          > the application doesn't store the state as it makes progress. My
          > hunch is
          > that making progress when outbound messages are not being delivered
          > isn't a
          > huge win most of the time - if there was work that could be
          > accomplished
          > without outbound network traffic, then it should be accomplished
          > regardless
          > of the messaging style.
          >
          >
          >> Nope, I mean make pub/sub more RESTful.
          >> Like I said, I don't know enough about REST principals to know if
          >> this is evil or not. I have a decent messaging middleware product
          >> today.
          >> I'm happy with it - I just want to know if its overkill or not or
          >> if there is a simpler way. If some of my integration types
          >> are best handled via JMS, fine with me. I just want to ask
          >> the "stupid question" - to make sure I'm not thinking JMS is
          >> better then it is.
          > JMS is fine. I have to ask though, how often do you use JMS to
          > integrate
          > with groups outside your control, or outside your company?
          >
          >>> Here is a sample app I wrote a while back that does what you are
          >>> talking
          > about :
          >>>
          > http://www.searchalert.net:8080/dierken/events/chat.html?topic=rest-
          > discuss
          >>> The underlying system holds messages here:
          > http://www.searchalert.net:8080/xmlrouter/msgs/rest-discuss
          >> So what is on the server - is there messaging behind it?
          > Nope, just a 50 line servlet and the messages in-memory in a linked
          > list.
          >
          >
          >
          >>> Are you sure you didn't mean N+1 responses?
          >> I'm talking about a client sending 1 request, having N
          >> services subscribe and send a response to the client.
          > I was kidding. The issue I was alluding to is - how do you know you've
          > received all the messages or if there is still one more straggler? The
          > pubsub approach is 'supposed' to be about loosely coupled systems,
          > but if
          > the publisher has to change to meet the specifications of each
          > consumer,
          > then that loose coupling has just tightened.
          >
        • Mike Dierken
          ... I did something like that about five years ago : http://sourceforge.net/projects/destiny That project is more of a (working) demo but the ideas are valid &
          Message 4 of 12 , May 1, 2006
          • 0 Attachment
            >
            > There is (well, will be) also a REST interface to the JMS
            > services. The REST service allows the creation of a service
            > request through a simple POST to a URI. The implementation of
            > this REST "service" connects to the JMS broker, sends the
            > corresponding message, and waits (with timeout) until it
            > receives confirmation that the message was received by the
            > JMS service. This confirmation of reception implies that the
            > service request has created a resource and this new resource
            > is returned to the REST client who can, at anytime, GET it to
            > see the status of his orignal request (running, completed,
            > cancelled, logging ...).
            I did something like that about five years ago :
            http://sourceforge.net/projects/destiny
            That project is more of a (working) demo but the ideas are valid & can be
            very useful.
          Your message has been successfully submitted and would be delivered to recipients shortly.