Loading ...
Sorry, an error occurred while loading the content.
 

Re: [decentralization] pull more robust than push?

Expand Messages
  • Wes Felter
    You say engineering is all about tradeoffs? I am shocked! :-) Wes Felter - wesley@felter.org - http://felter.org/wesley/
    Message 1 of 15 , Mar 15, 2004
      You say engineering is all about tradeoffs? I am shocked! :-)

      Wes Felter - wesley@... - http://felter.org/wesley/
    • Lindsey Smith
      ... (Trying to stick with Seth s set of assumptions), wouldn t Bloglines get crushed by the now-centralized update requests? Lindsey
      Message 2 of 15 , Mar 15, 2004
        Lucas Gonze wrote:
        > On the other hand, there is a way to save bandwidth that conserves the
        > client-driven nature of RSS -- use a centralized aggregator like
        > Bloglines.

        (Trying to stick with Seth's set of assumptions), wouldn't Bloglines get
        crushed by the now-centralized update requests?

        Lindsey
      • Lucas Gonze
        ... I don t think so, at least not in an ideal :) world. The user only has to hit Bloglines once to check all her feeds. If the average user has 100 subs
        Message 3 of 15 , Mar 15, 2004
          On Monday, Mar 15, 2004, at 16:41 America/New_York, Lindsey Smith wrote:

          > Lucas Gonze wrote:
          >> On the other hand, there is a way to save bandwidth that conserves the
          >> client-driven nature of RSS -- use a centralized aggregator like
          >> Bloglines.
          >
          > (Trying to stick with Seth's set of assumptions), wouldn't Bloglines
          > get
          > crushed by the now-centralized update requests?

          I don't think so, at least not in an ideal :) world. The user only has
          to hit Bloglines once to check all her feeds. If the average user has
          100 subs there's 1% of the original traffic. The nice thing is that
          this scales beautifully as average subs per user goes up.

          It could be a bigger improvement, but could be a lot worse too.
        • melc
          ... Couldn t you use some sort of swarming approach (like bittorrent) where, if possible, you try to get the updates from peers, rather than from the source?
          Message 4 of 15 , Mar 15, 2004
            On Mon, 15 Mar 2004, Seth Russell wrote:

            > 1) I want my client to be updated almost instantly when any one of a
            > very large number of RSS feeds changes.
            > 2) If my client polls every feed as fast as it can, then the servers of
            > popular feeds will be overwhelmed.
            >
            > So with those assumptions: how do i get what i want, still use client
            > side initiated events, not overwhelm servers, and still be able to scale
            > to millions of feeds and many millions of clients ?

            Couldn't you use some sort of swarming approach (like bittorrent) where,
            if possible, you try to get the updates from peers, rather than from the
            source?

            Ihor.
          • Justin Chapweske
            The answer is to do a meta-push. You build your system primarily around a pull-based design so that you gain the simplicity and robustness of the pull-based
            Message 5 of 15 , Mar 16, 2004
              The answer is to do a meta-push.

              You build your system primarily around a pull-based design so that you
              gain the simplicity and robustness of the pull-based approach.

              Then, you add an optional layer that allows receivers to register with
              the sources for an *unreliable* event notification that the source has
              been updated and should be polled again. By allowing the event
              notification to be unreliable, the system doesn't need to be concerned
              about dealing with failures during event notification.

              Registration for event notifications should be done on a leased basis,
              so the receiver will request a lease for some period of time, and if
              that lease is not renewed, then the source will forget about the
              receiver.

              As I've written here in the past, you can use mutually negotiated leases
              to allow the source to have the lease times be based on the expected
              uptime of the source such that the source does not need to store its
              event registration information durably. Thus, if the source crashes, it
              can simply rely on the fact that leases will be renewed periodically in
              order to repopulate its event registration list.

              All of this should be quite simple to do in a RESTful fashion.

              Now, an obvious limitation of this is the NAT/firewall problem of being
              able to send the meta-push notification back to the client.

              Well, with a decently designed system, this shouldn't be a problem since
              the receiver should be able to register a third-party service to receive
              its event notifications and simply perform a short interval poll on the
              third-party service. Each poll on the service would check ALL feeds
              that the receiver is interested in, thus amortizing the cost of the poll
              across all of the feeds.

              Now, this is starting to sound a lot like a third-party aggregation
              service, which I think will ultimately be the solution rather than all
              of this techno-mumbo-jumbo.

              By the way, can anyone point to cases where people are having actual
              problems with their RSS feeds being pegged too often?

              Thanks,

              -Justin

              > Yes, but assume the following (without challenge):
              >
              > 1) I want my client to be updated almost instantly when any one of a
              > very large number of RSS feeds changes.
              > 2) If my client polls every feed as fast as it can, then the servers of
              > popular feeds will be overwhelmed.
              >
              > So with those assumptions: how do i get what i want, still use client
              > side initiated events, not overwhelm servers, and still be able to scale
              > to millions of feeds and many millions of clients ?
              >
              > Seth
              >
              >
              >
              >
              > Announce or discover P2P conferences on the P2P Conference Wiki at
              > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferenceshe P2P Conference Wiki at
              > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
              > Yahoo! Groups Links
              >
              >
              >
              >
            • Julian Bond
              ... Slashdot. Ironically. -- Julian Bond Email&MSM: julian.bond at voidstar.com Webmaster: http://www.ecademy.com/ Personal WebLog:
              Message 6 of 15 , Mar 16, 2004
                Justin Chapweske <justin@...> wrote:
                >By the way, can anyone point to cases where people are having actual
                >problems with their RSS feeds being pegged too often?

                Slashdot. Ironically.

                --
                Julian Bond Email&MSM: julian.bond at voidstar.com
                Webmaster: http://www.ecademy.com/
                Personal WebLog: http://www.voidstar.com/
                M: +44 (0)77 5907 2173 T: +44 (0)192 0412 433
              • Lucas Gonze
                On Tuesday, Mar 16, 2004, at 13:00 America/New_York, Justin Chapweske ... This pretty well describes the architecture of RSS pings.
                Message 7 of 15 , Mar 16, 2004
                  On Tuesday, Mar 16, 2004, at 13:00 America/New_York, Justin Chapweske
                  wrote:
                  > You build your system primarily around a pull-based design so that you
                  > gain the simplicity and robustness of the pull-based approach.
                  >
                  > Then, you add an optional layer that allows receivers to register with
                  > the sources for an *unreliable* event notification that the source has
                  > been updated and should be polled again. By allowing the event
                  > notification to be unreliable, the system doesn't need to be concerned
                  > about dealing with failures during event notification.

                  This pretty well describes the architecture of RSS pings.
                Your message has been successfully submitted and would be delivered to recipients shortly.