Loading ...
Sorry, an error occurred while loading the content.

Re: [decentralization] pull more robust than push?

Expand Messages
  • Wes Felter
    You say engineering is all about tradeoffs? I am shocked! :-) Wes Felter - wesley@felter.org - http://felter.org/wesley/
    Message 1 of 15 , Mar 15, 2004
    • 0 Attachment
      You say engineering is all about tradeoffs? I am shocked! :-)

      Wes Felter - wesley@... - http://felter.org/wesley/
    • Lucas Gonze
      ... I don t think so, at least not in an ideal :) world. The user only has to hit Bloglines once to check all her feeds. If the average user has 100 subs
      Message 2 of 15 , Mar 15, 2004
      • 0 Attachment
        On Monday, Mar 15, 2004, at 16:41 America/New_York, Lindsey Smith wrote:

        > Lucas Gonze wrote:
        >> On the other hand, there is a way to save bandwidth that conserves the
        >> client-driven nature of RSS -- use a centralized aggregator like
        >> Bloglines.
        >
        > (Trying to stick with Seth's set of assumptions), wouldn't Bloglines
        > get
        > crushed by the now-centralized update requests?

        I don't think so, at least not in an ideal :) world. The user only has
        to hit Bloglines once to check all her feeds. If the average user has
        100 subs there's 1% of the original traffic. The nice thing is that
        this scales beautifully as average subs per user goes up.

        It could be a bigger improvement, but could be a lot worse too.
      • melc
        ... Couldn t you use some sort of swarming approach (like bittorrent) where, if possible, you try to get the updates from peers, rather than from the source?
        Message 3 of 15 , Mar 15, 2004
        • 0 Attachment
          On Mon, 15 Mar 2004, Seth Russell wrote:

          > 1) I want my client to be updated almost instantly when any one of a
          > very large number of RSS feeds changes.
          > 2) If my client polls every feed as fast as it can, then the servers of
          > popular feeds will be overwhelmed.
          >
          > So with those assumptions: how do i get what i want, still use client
          > side initiated events, not overwhelm servers, and still be able to scale
          > to millions of feeds and many millions of clients ?

          Couldn't you use some sort of swarming approach (like bittorrent) where,
          if possible, you try to get the updates from peers, rather than from the
          source?

          Ihor.
        • Justin Chapweske
          The answer is to do a meta-push. You build your system primarily around a pull-based design so that you gain the simplicity and robustness of the pull-based
          Message 4 of 15 , Mar 16, 2004
          • 0 Attachment
            The answer is to do a meta-push.

            You build your system primarily around a pull-based design so that you
            gain the simplicity and robustness of the pull-based approach.

            Then, you add an optional layer that allows receivers to register with
            the sources for an *unreliable* event notification that the source has
            been updated and should be polled again. By allowing the event
            notification to be unreliable, the system doesn't need to be concerned
            about dealing with failures during event notification.

            Registration for event notifications should be done on a leased basis,
            so the receiver will request a lease for some period of time, and if
            that lease is not renewed, then the source will forget about the
            receiver.

            As I've written here in the past, you can use mutually negotiated leases
            to allow the source to have the lease times be based on the expected
            uptime of the source such that the source does not need to store its
            event registration information durably. Thus, if the source crashes, it
            can simply rely on the fact that leases will be renewed periodically in
            order to repopulate its event registration list.

            All of this should be quite simple to do in a RESTful fashion.

            Now, an obvious limitation of this is the NAT/firewall problem of being
            able to send the meta-push notification back to the client.

            Well, with a decently designed system, this shouldn't be a problem since
            the receiver should be able to register a third-party service to receive
            its event notifications and simply perform a short interval poll on the
            third-party service. Each poll on the service would check ALL feeds
            that the receiver is interested in, thus amortizing the cost of the poll
            across all of the feeds.

            Now, this is starting to sound a lot like a third-party aggregation
            service, which I think will ultimately be the solution rather than all
            of this techno-mumbo-jumbo.

            By the way, can anyone point to cases where people are having actual
            problems with their RSS feeds being pegged too often?

            Thanks,

            -Justin

            > Yes, but assume the following (without challenge):
            >
            > 1) I want my client to be updated almost instantly when any one of a
            > very large number of RSS feeds changes.
            > 2) If my client polls every feed as fast as it can, then the servers of
            > popular feeds will be overwhelmed.
            >
            > So with those assumptions: how do i get what i want, still use client
            > side initiated events, not overwhelm servers, and still be able to scale
            > to millions of feeds and many millions of clients ?
            >
            > Seth
            >
            >
            >
            >
            > Announce or discover P2P conferences on the P2P Conference Wiki at
            > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferenceshe P2P Conference Wiki at
            > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
            > Yahoo! Groups Links
            >
            >
            >
            >
          • Julian Bond
            ... Slashdot. Ironically. -- Julian Bond Email&MSM: julian.bond at voidstar.com Webmaster: http://www.ecademy.com/ Personal WebLog:
            Message 5 of 15 , Mar 16, 2004
            • 0 Attachment
              Justin Chapweske <justin@...> wrote:
              >By the way, can anyone point to cases where people are having actual
              >problems with their RSS feeds being pegged too often?

              Slashdot. Ironically.

              --
              Julian Bond Email&MSM: julian.bond at voidstar.com
              Webmaster: http://www.ecademy.com/
              Personal WebLog: http://www.voidstar.com/
              M: +44 (0)77 5907 2173 T: +44 (0)192 0412 433
            • Lucas Gonze
              On Tuesday, Mar 16, 2004, at 13:00 America/New_York, Justin Chapweske ... This pretty well describes the architecture of RSS pings.
              Message 6 of 15 , Mar 16, 2004
              • 0 Attachment
                On Tuesday, Mar 16, 2004, at 13:00 America/New_York, Justin Chapweske
                wrote:
                > You build your system primarily around a pull-based design so that you
                > gain the simplicity and robustness of the pull-based approach.
                >
                > Then, you add an optional layer that allows receivers to register with
                > the sources for an *unreliable* event notification that the source has
                > been updated and should be polled again. By allowing the event
                > notification to be unreliable, the system doesn't need to be concerned
                > about dealing with failures during event notification.

                This pretty well describes the architecture of RSS pings.
              Your message has been successfully submitted and would be delivered to recipients shortly.