Loading ...
Sorry, an error occurred while loading the content.

atom pub/high usage

Expand Messages
  • Greg Young
    Its pretty well known how to do a quick pub/sub with atom pub. If we get loads and loads of consumers though it might be beneficial to say hang a http
    Message 1 of 9 , Feb 6, 2012
    • 0 Attachment
      Its pretty well known how to do a quick pub/sub with atom pub. If we
      get loads and loads of consumers though it might be beneficial to say
      hang a http connection and push results down the pipe in near real
      time as opposed to polling (performance optimization only). This can
      be done in a generic way, does anyone happen to know of a system that
      just does that out of the box as opposed to building it?

      --
      Le doute n'est pas une condition agréable, mais la certitude est absurde.
    • Jan Algermissen
      ... Huh? Are you saying, you want to switch from polling to push when number of users grow? I d suggest that, the more number of users you have, the less is
      Message 2 of 9 , Feb 6, 2012
      • 0 Attachment
        On Feb 6, 2012, at 2:36 PM, Greg Young wrote:

        > Its pretty well known how to do a quick pub/sub with atom pub. If we
        > get loads and loads of consumers though it might be beneficial to say
        > hang a http connection

        Huh? Are you saying, you want to switch from polling to push when number of users grow?

        I'd suggest that, the more number of users you have, the less is push a good architectural choice. Better to make use of HTTP caching to decrease umber of requests that actually hit your backend.

        Look at archived feeds for this purpose <http://tools.ietf.org/html/rfc5005> or check out <http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons> which I think is indeed a very practical solution despite its 'esoteric' appeal at first glance :-)

        If you feel you must use push, then there are solutions already such as XMPP.


        Jan


        > and push results down the pipe in near real
        > time as opposed to polling (performance optimization only). This can
        > be done in a generic way, does anyone happen to know of a system that
        > just does that out of the box as opposed to building it?
        >
        > --
        > Le doute n'est pas une condition agréable, mais la certitude est absurde.
        >
      • Greg Young
        There is good reason to switch to push as FREQUENCY of messages gets higher. Try polling the twitter stream. On Mon, Feb 6, 2012 at 8:48 AM, Jan Algermissen
        Message 3 of 9 , Feb 6, 2012
        • 0 Attachment
          There is good reason to switch to push as FREQUENCY of messages gets
          higher. Try polling the twitter stream.

          On Mon, Feb 6, 2012 at 8:48 AM, Jan Algermissen
          <jan.algermissen@...> wrote:
          >
          > On Feb 6, 2012, at 2:36 PM, Greg Young wrote:
          >
          >> Its pretty well known how to do a quick pub/sub with atom pub. If we
          >> get loads and loads of consumers though it might be beneficial to say
          >> hang a http connection
          >
          > Huh? Are you saying, you want to switch from polling to push when number of users grow?
          >
          > I'd suggest that, the more number of users you have, the less is push a good architectural choice. Better to make use of HTTP caching to decrease umber of requests that actually hit your backend.
          >
          > Look at archived feeds for this purpose <http://tools.ietf.org/html/rfc5005> or check out <http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons> which I think is indeed a very practical solution despite its 'esoteric' appeal at first glance :-)
          >
          > If you feel you must use push, then there are solutions already such as XMPP.
          >
          >
          > Jan
          >
          >
          >> and push results down the pipe in near real
          >> time as opposed to polling (performance optimization only). This can
          >> be done in a generic way, does anyone happen to know of a system that
          >> just does that out of the box as opposed to building it?
          >>
          >> --
          >> Le doute n'est pas une condition agréable, mais la certitude est absurde.
          >>
          >



          --
          Le doute n'est pas une condition agréable, mais la certitude est absurde.
        • Greg Young
          My bad for not being clear in the original post. ... -- Le doute n est pas une condition agréable, mais la certitude est absurde.
          Message 4 of 9 , Feb 6, 2012
          • 0 Attachment
            My bad for not being clear in the original post.

            On Mon, Feb 6, 2012 at 8:51 AM, Greg Young <gregoryyoung1@...> wrote:
            > There is good reason to switch to push as FREQUENCY of messages gets
            > higher. Try polling the twitter stream.
            >
            > On Mon, Feb 6, 2012 at 8:48 AM, Jan Algermissen
            > <jan.algermissen@...> wrote:
            >>
            >> On Feb 6, 2012, at 2:36 PM, Greg Young wrote:
            >>
            >>> Its pretty well known how to do a quick pub/sub with atom pub. If we
            >>> get loads and loads of consumers though it might be beneficial to say
            >>> hang a http connection
            >>
            >> Huh? Are you saying, you want to switch from polling to push when number of users grow?
            >>
            >> I'd suggest that, the more number of users you have, the less is push a good architectural choice. Better to make use of HTTP caching to decrease umber of requests that actually hit your backend.
            >>
            >> Look at archived feeds for this purpose <http://tools.ietf.org/html/rfc5005> or check out <http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons> which I think is indeed a very practical solution despite its 'esoteric' appeal at first glance :-)
            >>
            >> If you feel you must use push, then there are solutions already such as XMPP.
            >>
            >>
            >> Jan
            >>
            >>
            >>> and push results down the pipe in near real
            >>> time as opposed to polling (performance optimization only). This can
            >>> be done in a generic way, does anyone happen to know of a system that
            >>> just does that out of the box as opposed to building it?
            >>>
            >>> --
            >>> Le doute n'est pas une condition agréable, mais la certitude est absurde.
            >>>
            >>
            >
            >
            >
            > --
            > Le doute n'est pas une condition agréable, mais la certitude est absurde.



            --
            Le doute n'est pas une condition agréable, mais la certitude est absurde.
          • Jan Algermissen
            ... Well, try pushing the Twitter stream to zillions of clients :-) What I was referring to is to split the stream in archived (==static) pages that are then
            Message 5 of 9 , Feb 6, 2012
            • 0 Attachment
              On Feb 6, 2012, at 2:51 PM, Greg Young wrote:

              > There is good reason to switch to push as FREQUENCY of messages gets
              > higher. Try polling the twitter stream.

              Well, try pushing the Twitter stream to zillions of clients :-)

              What I was referring to is to split the stream in archived (==static) pages that are then cacheable.

              Jan


              >
              > On Mon, Feb 6, 2012 at 8:48 AM, Jan Algermissen
              > <jan.algermissen@...> wrote:
              >>
              >> On Feb 6, 2012, at 2:36 PM, Greg Young wrote:
              >>
              >>> Its pretty well known how to do a quick pub/sub with atom pub. If we
              >>> get loads and loads of consumers though it might be beneficial to say
              >>> hang a http connection
              >>
              >> Huh? Are you saying, you want to switch from polling to push when number of users grow?
              >>
              >> I'd suggest that, the more number of users you have, the less is push a good architectural choice. Better to make use of HTTP caching to decrease umber of requests that actually hit your backend.
              >>
              >> Look at archived feeds for this purpose <http://tools.ietf.org/html/rfc5005> or check out <http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons> which I think is indeed a very practical solution despite its 'esoteric' appeal at first glance :-)
              >>
              >> If you feel you must use push, then there are solutions already such as XMPP.
              >>
              >>
              >> Jan
              >>
              >>
              >>> and push results down the pipe in near real
              >>> time as opposed to polling (performance optimization only). This can
              >>> be done in a generic way, does anyone happen to know of a system that
              >>> just does that out of the box as opposed to building it?
              >>>
              >>> --
              >>> Le doute n'est pas une condition agréable, mais la certitude est absurde.
              >>>
              >>
              >
              >
              >
              > --
              > Le doute n'est pas une condition agréable, mais la certitude est absurde.
              >
              >
              > ------------------------------------
              >
              > Yahoo! Groups Links
              >
              >
              >
            • Kevin Duffey
              Hi all, I am trying to do the best I can for our API needs to avoid making some REST calls basically be RPC calls. I can t explain in detail regarding the
              Message 6 of 9 , Feb 7, 2012
              • 0 Attachment
                Hi all,

                I am trying to do the best I can for our API needs to avoid making some REST calls basically be RPC calls. I can't explain in detail regarding the company I work for as we're still in stealth/startup mode so I'll do the best I can to make sense.

                Our API uses HtmlUnit (Java) to "scrape" some web sites. Basically we pull down a sites login page, try some credentials that the consumer passed to us, and attempt to log a user in. It sounds fishy.. but we're really not trying to randomly log in to sites and steal credentials. The consumer of our API will ask their end users for credentials, pass those to us via a REST call and we'll attempt to validate the credentials are good, sending back a response to the consumer that they are good or not. However, as far as I can tell this does not modify a resource, or create a new resource on our server side. It's basically a proxy for a consumer app to use our API to try validating user credentials for different services. The consumer gets the benefit of being able to avoid having to do the page scraping and/or using other service APIs..they just use our one API and specify the service and pass the credentials.

                The problem is.. we cant use GET since you can't pass a body. I thought of query params for the creds, but that seems hacky to me. So PUT seems the logical method to use, passing in credentials (possibly other values too) in the body. Only, as I said above, we're not storing these values, changing any data on our API server side.. we're merely checking to see if what was sent to our API works at the service selected (lets say yahoo, gmail, etc). So I guess you could say the boolean flag of whether or not the creds worked changes..but that's not really the resource.

                So I am asking you experts.. how is something like this done via REST? I know.. from all the posts I've read the past few years, I fit that mold of those trying to turn an RPC into REST. Essentially if the community was "ok" with using RPC like syntax with the ease of REST calls, I would be ok with this. Because it seems a lot of the community balks at using REST in lieu of SOAP and other RPC mechanisms, I am left to figuring out if there is some way to do this with REST.

                Feel free to slap me around a bit if I am completely missing some key points about using REST. I think I have a pretty good understanding of it, but no doubt there is room to learn.

                Thank you.




              • mike amundsen
                Kevin: not sure if this helps, but regarding the as far as I can tell this does not modify a resource, or create a new resource line, i ll share this
                Message 7 of 9 , Feb 7, 2012
                • 0 Attachment
                  Kevin:

                  not sure if this helps, but regarding the " as far as I can tell this does not modify a resource, or create a new resource" line, i'll share this observation:

                  First, the POST/PUT actions are _not_ reserved for creating/modifying some concrete entity in your server's problem domain. You can use these idempotent(PUT) and non-idempotent(POST) actions to send data bodies to the server for processing of all kinds including comparing the data against data on the server, logging for later use, etc.

                  I often "model" actions such as "login", "ticket request", "queue up a job", etc. as a "write-record" pattern. IOW, if you attempt to log into the server, I might log the action (date, time, username, other context data) for later audit review. When I do this, it makes good sense to use a POST w/ body to send the data. These actions often result in a new "resource" which hold the results of the attempt ("here's your ticket", "you job has been added to the queue", etc.) and sometimes this resource will be updated/refreshed over the life of the job, etc.

                  I think that would fit your scenario just fine. you need not save all the data sent by the user (i.e. password, etc.)


                  mca
                  http://amundsen.com/blog/
                  http://twitter.com@mamund
                  http://mamund.com/foaf.rdf#me




                  On Tue, Feb 7, 2012 at 14:47, Kevin Duffey <andjarnic@...> wrote:


                  Hi all,

                  I am trying to do the best I can for our API needs to avoid making some REST calls basically be RPC calls. I can't explain in detail regarding the company I work for as we're still in stealth/startup mode so I'll do the best I can to make sense.

                  Our API uses HtmlUnit (Java) to "scrape" some web sites. Basically we pull down a sites login page, try some credentials that the consumer passed to us, and attempt to log a user in. It sounds fishy.. but we're really not trying to randomly log in to sites and steal credentials. The consumer of our API will ask their end users for credentials, pass those to us via a REST call and we'll attempt to validate the credentials are good, sending back a response to the consumer that they are good or not. However, as far as I can tell this does not modify a resource, or create a new resource on our server side. It's basically a proxy for a consumer app to use our API to try validating user credentials for different services. The consumer gets the benefit of being able to avoid having to do the page scraping and/or using other service APIs..they just use our one API and specify the service and pass the credentials.

                  The problem is.. we cant use GET since you can't pass a body. I thought of query params for the creds, but that seems hacky to me. So PUT seems the logical method to use, passing in credentials (possibly other values too) in the body. Only, as I said above, we're not storing these values, changing any data on our API server side.. we're merely checking to see if what was sent to our API works at the service selected (lets say yahoo, gmail, etc). So I guess you could say the boolean flag of whether or not the creds worked changes..but that's not really the resource.

                  So I am asking you experts.. how is something like this done via REST? I know.. from all the posts I've read the past few years, I fit that mold of those trying to turn an RPC into REST. Essentially if the community was "ok" with using RPC like syntax with the ease of REST calls, I would be ok with this. Because it seems a lot of the community balks at using REST in lieu of SOAP and other RPC mechanisms, I am left to figuring out if there is some way to do this with REST.

                  Feel free to slap me around a bit if I am completely missing some key points about using REST. I think I have a pretty good understanding of it, but no doubt there is room to learn.

                  Thank you.







                • Kevin Duffey
                      Thank you for the replies. Both you and Ben seem to be saying the same thing.. some sort of job queue resource. I probably should have added that all of
                  Message 8 of 9 , Feb 7, 2012
                  • 0 Attachment
                        Thank you for the replies. Both you and Ben seem to be saying the same thing.. some sort of job queue resource. I probably should have added that all of our API calls other than GETs are async. They return immediately.. so essentially our API becomes one big job queue. We also support callbacks. Consumers register a URL with us that we then send notifications back to. We will eventually support polling as well, and in doing so we'll need to respond with resource IDs, and in some cases we do so that when we send notifications via the callback to them, they can match up the request we make to their callback handler (with body) with the ID that they were given when they made the async request to our API. IOW, some of our resources create a resource to generate the id, send that back right away, then process the bulk of the request in an EJB 3 async method. Some of our calls will require possibly several seconds of time to process for one user and we don't want to tie up the pipes.. some of them may even take minutes to days (again..can't provide the details I wish I could..but it's possible they might take some time before a response could be given back to the consumer). Think of it as an IM.. I can send it to you and you could respond almost immediately if you are there..but if you're not, you may respond with an auto-reply immediately then send me an email later with more details. Anyway, I do like the idea of a job queue of sorts.. primarily because it gives an actual resource ID a consumer can then use to poll, and more so in our callback scenario, associate with to a callback we send with an id they can use to match to one they were given when they sent the initial request to us. We don't do that right now with this particular scenario, so that would work well here.




                    From: mike amundsen <mamund@...>
                    To: Kevin Duffey <andjarnic@...>
                    Cc: Rest List <rest-discuss@yahoogroups.com>
                    Sent: Tuesday, February 7, 2012 12:15 PM
                    Subject: Re: [rest-discuss] How to make RPC-like call RESTfully

                    Kevin:

                    not sure if this helps, but regarding the " as far as I can tell this does not modify a resource, or create a new resource" line, i'll share this observation:

                    First, the POST/PUT actions are _not_ reserved for creating/modifying some concrete entity in your server's problem domain. You can use these idempotent(PUT) and non-idempotent(POST) actions to send data bodies to the server for processing of all kinds including comparing the data against data on the server, logging for later use, etc.

                    I often "model" actions such as "login", "ticket request", "queue up a job", etc. as a "write-record" pattern. IOW, if you attempt to log into the server, I might log the action (date, time, username, other context data) for later audit review. When I do this, it makes good sense to use a POST w/ body to send the data. These actions often result in a new "resource" which hold the results of the attempt ("here's your ticket", "you job has been added to the queue", etc.) and sometimes this resource will be updated/refreshed over the life of the job, etc.

                    I think that would fit your scenario just fine. you need not save all the data sent by the user (i.e. password, etc.)


                    mca
                    http://amundsen.com/blog/
                    http://twitter.com@mamund
                    http://mamund.com/foaf.rdf#me




                    On Tue, Feb 7, 2012 at 14:47, Kevin Duffey <andjarnic@...> wrote:


                    Hi all,

                    I am trying to do the best I can for our API needs to avoid making some REST calls basically be RPC calls. I can't explain in detail regarding the company I work for as we're still in stealth/startup mode so I'll do the best I can to make sense.

                    Our API uses HtmlUnit (Java) to "scrape" some web sites. Basically we pull down a sites login page, try some credentials that the consumer passed to us, and attempt to log a user in. It sounds fishy.. but we're really not trying to randomly log in to sites and steal credentials. The consumer of our API will ask their end users for credentials, pass those to us via a REST call and we'll attempt to validate the credentials are good, sending back a response to the consumer that they are good or not. However, as far as I can tell this does not modify a resource, or create a new resource on our server side. It's basically a proxy for a consumer app to use our API to try validating user credentials for different services. The consumer gets the benefit of being able to avoid having to do the page scraping and/or using other service APIs..they just use our one API and specify the service and pass the credentials.

                    The problem is.. we cant use GET since you can't pass a body. I thought of query params for the creds, but that seems hacky to me. So PUT seems the logical method to use, passing in credentials (possibly other values too) in the body. Only, as I said above, we're not storing these values, changing any data on our API server side.. we're merely checking to see if what was sent to our API works at the service selected (lets say yahoo, gmail, etc). So I guess you could say the boolean flag of whether or not the creds worked changes..but that's not really the resource.

                    So I am asking you experts.. how is something like this done via REST? I know.. from all the posts I've read the past few years, I fit that mold of those trying to turn an RPC into REST. Essentially if the community was "ok" with using RPC like syntax with the ease of REST calls, I would be ok with this. Because it seems a lot of the community balks at using REST in lieu of SOAP and other RPC mechanisms, I am left to figuring out if there is some way to do this with REST.

                    Feel free to slap me around a bit if I am completely missing some key points about using REST. I think I have a pretty good understanding of it, but no doubt there is room to learn.

                    Thank you.









                  • Sebastien Lambla
                    Quick two points on this: ID is already part of the application protocl, its the URI. If you model it independently, you may be introducing coupling back to
                    Message 9 of 9 , Feb 7, 2012
                    • 0 Attachment
                       Quick two points on this:

                      ID is already part of the application protocl, its' the URI. If you  model it independently, you may be introducing coupling back to a second identification constraint in the protocl that is not part of rest.

                      As for alternatives to the client-server pull-based interaction mechanisms that are part of ReST, there's a lot of interesting architectural discussions to be had on those, and sometimes the benefits do outweight the costs. Be aware however that those are not part of the ReSTful architectural style, so any effort to discuss those would probably provoke a lot of positive conversation if you introduce them as alternatives / different / cost-benefit analysis. You can still design async work with HTTP and still get leverage out of *that* app protocol, and you can also build *something else* that fits the same requirements in an architecture that doesn't conform to ReST. As long as you're clear on the boundary within the two, I'm sure any of us architect devs / rationale people will happily discuss those alternatives and what positives and negatives they present.

                      Seb



                      From: rest-discuss@yahoogroups.com [rest-discuss@yahoogroups.com] on behalf of Kevin Duffey [andjarnic@...]
                      Sent: 07 February 2012 21:12
                      To: Rest List
                      Subject: Re: [rest-discuss] How to make RPC-like call RESTfully



                          Thank you for the replies. Both you and Ben seem to be saying the same thing.. some sort of job queue resource. I probably should have added that all of our API calls other than GETs are async. They return immediately.. so essentially our API becomes one big job queue. We also support callbacks. Consumers register a URL with us that we then send notifications back to. We will eventually support polling as well, and in doing so we'll need to respond with resource IDs, and in some cases we do so that when we send notifications via the callback to them, they can match up the request we make to their callback handler (with body) with the ID that they were given when they made the async request to our API. IOW, some of our resources create a resource to generate the id, send that back right away, then process the bulk of the request in an EJB 3 async method. Some of our calls will require possibly several seconds of time to process for one user and we don't want to tie up the pipes.. some of them may even take minutes to days (again..can't provide the details I wish I could..but it's possible they might take some time before a response could be given back to the consumer). Think of it as an IM.. I can send it to you and you could respond almost immediately if you are there..but if you're not, you may respond with an auto-reply immediately then send me an email later with more details. Anyway, I do like the idea of a job queue of sorts.. primarily because it gives an actual resource ID a consumer can then use to poll, and more so in our callback scenario, associate with to a callback we send with an id they can use to match to one they were given when they sent the initial request to us. We don't do that right now with this particular scenario, so that would work well here.




                      From: mike amundsen <mamund@...>
                      To: Kevin Duffey <andjarnic@...>
                      Cc: Rest List <rest-discuss@yahoogroups.com>
                      Sent: Tuesday, February 7, 2012 12:15 PM
                      Subject: Re: [rest-discuss] How to make RPC-like call RESTfully

                      Kevin:

                      not sure if this helps, but regarding the " as far as I can tell this does not modify a resource, or create a new resource" line, i'll share this observation:

                      First, the POST/PUT actions are _not_ reserved for creating/modifying some concrete entity in your server's problem domain. You can use these idempotent(PUT) and non-idempotent(POST) actions to send data bodies to the server for processing of all kinds including comparing the data against data on the server, logging for later use, etc.

                      I often "model" actions such as "login", "ticket request", "queue up a job", etc. as a "write-record" pattern. IOW, if you attempt to log into the server, I might log the action (date, time, username, other context data) for later audit review. When I do this, it makes good sense to use a POST w/ body to send the data. These actions often result in a new "resource" which hold the results of the attempt ("here's your ticket", "you job has been added to the queue", etc.) and sometimes this resource will be updated/refreshed over the life of the job, etc.

                      I think that would fit your scenario just fine. you need not save all the data sent by the user (i.e. password, etc.)


                      mca
                      http://amundsen.com/blog/
                      http://twitter.com@mamund
                      http://mamund.com/foaf.rdf#me




                      On Tue, Feb 7, 2012 at 14:47, Kevin Duffey <andjarnic@...> wrote:


                      Hi all,

                      I am trying to do the best I can for our API needs to avoid making some REST calls basically be RPC calls. I can't explain in detail regarding the company I work for as we're still in stealth/startup mode so I'll do the best I can to make sense.

                      Our API uses HtmlUnit (Java) to "scrape" some web sites. Basically we pull down a sites login page, try some credentials that the consumer passed to us, and attempt to log a user in. It sounds fishy.. but we're really not trying to randomly log in to sites and steal credentials. The consumer of our API will ask their end users for credentials, pass those to us via a REST call and we'll attempt to validate the credentials are good, sending back a response to the consumer that they are good or not. However, as far as I can tell this does not modify a resource, or create a new resource on our server side. It's basically a proxy for a consumer app to use our API to try validating user credentials for different services. The consumer gets the benefit of being able to avoid having to do the page scraping and/or using other service APIs..they just use our one API and specify the service and pass the credentials.

                      The problem is.. we cant use GET since you can't pass a body. I thought of query params for the creds, but that seems hacky to me. So PUT seems the logical method to use, passing in credentials (possibly other values too) in the body. Only, as I said above, we're not storing these values, changing any data on our API server side.. we're merely checking to see if what was sent to our API works at the service selected (lets say yahoo, gmail, etc). So I guess you could say the boolean flag of whether or not the creds worked changes..but that's not really the resource.

                      So I am asking you experts.. how is something like this done via REST? I know.. from all the posts I've read the past few years, I fit that mold of those trying to turn an RPC into REST. Essentially if the community was "ok" with using RPC like syntax with the ease of REST calls, I would be ok with this. Because it seems a lot of the community balks at using REST in lieu of SOAP and other RPC mechanisms, I am left to figuring out if there is some way to do this with REST.

                      Feel free to slap me around a bit if I am completely missing some key points about using REST. I think I have a pretty good understanding of it, but no doubt there is room to learn.

                      Thank you.











                    Your message has been successfully submitted and would be delivered to recipients shortly.