Loading ...
Sorry, an error occurred while loading the content.
 

Dynamic Networking

Expand Messages
  • Justin Chapweske
    ... I m CC ing this message to decentralization and p2p-hackers to engage the larger community on this interesting topic. James, I heartily agree with your
    Message 1 of 12 , Apr 22, 2004
      > I have absolutely no problem with HTTP, being one of the "team of two
      > plus a
      > few" who released Tomcat to the world in '99. The problem I have with
      > the
      > HTTP application programming model is entirely in the realm of
      > deployment.
      > Server as largely assumed to be reachible via static addresses, IP. My
      > interests
      > are in "everything is a server, everthing is a client" such that
      > servers can
      > be instantiated, provissioned, and reclaimed dynamically w/ no harm
      > and in
      > fact "to great advantage" to the application network. In this model
      > the app
      > developer is literally constrained as to what the deployment network
      > looks
      > like. With JXTA, the app developer *owns* the network, can change it
      > as the
      > need arises, etc. BTW, this is the very epiphany I had that clued me
      > in that
      > there had to be a better way moving forward.

      I'm CC'ing this message to decentralization and p2p-hackers to engage
      the larger community on this interesting topic.

      James, I heartily agree with your goal of transcending the limitations
      of static addresses and allowing the network to be much more dynamic.

      However, I think we have different approaches for achieving this goal.

      My approach is to treat dynamic networking as a naming or URI problem.
      JXTA appears to take the approach that dynamic networking is a routing
      or messaging problem.

      Now, there are multiple levels at which one can solve dynamic networking
      from a naming perspective. The first level corresponds to solving the
      problem at the IP level using multicast and anycast addresses. A
      wonderful example of this is zeroconf/rendezvous. By allowing URIs/IP
      addresses to be resolved to any valid machine on the LAN, transparent
      dynamic networking is now enabled by any application that sits on top of
      IP and DNS.

      Of course, the limitation of dynamic networking at the IP level is that
      it only works for "anycast" type services where a number of nodes can
      provide the desired service. But, it does not provide for
      point-to-point dynamic networking services where a single specific node
      (possibly roaming) can provide the desired service.

      To solve the issue of contacting a single, possibly roaming, node with
      no reliable static IP address, we can look at the next layer up the
      naming stack - DNS. To solve dynamic networking at the DNS level, we
      make use of two different approaches: Akamai-style request routing via
      DNS, and dynamic DNS that routes to a single roaming node. The Akamai
      approach provides the anycast type functionality where a single name may
      resolve to any number of valid endpoints that can facilitate the desired
      service. The dynamic DNS approach allows a single roaming node to be
      contacted via a sturdy identifier where ever it may be.

      Now, the main limitations of the DNS approach is the reachability of
      nodes that are behind NAT/Firewalls and are unroutable via IP and the
      centralization of DNS infrastructure. The first issue can be solved by
      something like IPv6/Teredo or simply introducing public rendezvous
      points that proxy TCP streams to hidden nodes. This could be easily
      accomplished with SSH port forwarding. The second problem of
      centralization of DNS infrastructure is probably not an issue for most
      applications since all you need is a single static server to handle the
      DNS serving, and you can easily have a fully decentralized name space
      underneath your top level registered domain.

      Now, assuming that the NAT and centralization limitations of DNS are too
      limiting for your application, or you simply don't feel comfortable
      developing an application at the DNS level, the next layer in the naming
      stack is URIs.

      In order to provide dynamic networking at the URI naming level, you
      simply need to switch a portion of your URIs to non-DNS-based URIs.
      There is no need to switch all of your URIs to be location independent,
      and indeed in Swarmcast we freely mix static HTTP URIs with location
      independent UUID URIs.

      Location independent URIs, by their very nature, imply support for both
      an anycast model, as well as a single-end-node or point-to-point model.
      The only difference between these two cases is the number of
      authoritative sources that can provide the service associated with the
      URI in a secure fashion. In an anycast model, a number of nodes can
      authoritatively serve a URI, and in a point-to-point model, only a
      single end node can authoritatively serve a URI.

      Now, even though these location-independent URIs are not HTTP URIs, it
      doesn't mean that they can't still be served via HTTP. There are two
      ways to serve a non-HTTP URI via HTTP, the first is by proxying, and the
      second is by providing explicit URI resolution services. I'll skip over
      proxying for now and focus on URI resolution services.

      URI resolution services are very straight-forward. They are basically a
      mechanism to discover other equivalent URIs for a given URI. The goal
      of URI resolution in the context of dynamic networking is to resolve a
      location independent URI into one or more authoritative HTTP URIs. This
      concept is laid out in THTTP/RFC 2169 and the Content-Addressable Web
      (http://open-content.net/specs/draft-jchapweske-caw-03.html), though the
      latter is in severe need of updating/refinement.

      Under the covers, there are numerous ways that one can implement URI
      resolution, but I believe one should be careful to hide the details of
      the URI resolution specifics and allow developers to focus on simply
      naming and consuming resources.

      For example:

      o Gnutella provides a form of URI resolution by mapping keywords to
      URNs, then using THTTP to further resolve to specific HTTP end points.
      With magnet-URIs, people have done exactly that.

      o DHT's are an obvious candidate for URI resolution under the covers - I
      would love to see a DHT that provides a simple RFC 2169/THTTP interface
      for looking up URIs.

      o JXTA could also be used under the covers for URI resolution, using its
      security and group communication features as a value add. Once the URI
      is resolved, the end-user application would establish a direct
      connection to the resulting URI(s) to finish the transaction.

      Now, the last piece of this whole thing is HTTP proxying and caching.
      Now, even after mentioning earlier how DNS/HTTP-based URIs may have
      certain limitations, this all goes out the window when you consider HTTP
      caching semantics.

      HTTP defines very clear semantics for allowing a non-point-of-origin
      node to service a request directly and it can also fulfill requests for
      non-HTTP URIs. This allows fully dynamic URI-based networking to take
      place hidden completely behind the abstraction of an HTTP proxy. This
      approach also allows additional features like automatic-failover to be
      automatically added to applications w/o any modification to the original
      application

      My point behind this is that routing and messaging are not the answer to
      facilitating dynamic networking. An end application should not need to
      be concerned with how to route a message to a dynamic receiver, nor
      should it be concerned with protocols at a higher level than absolutely
      necessary - If you need dynamic networking on a LAN, use zeroconf. If
      you need dynamic networking among publicly addressable servers, use
      DNS. If you need dynamic networking to NAT'd nodes, then use URIs and
      HTTP. No matter which route you take, the code in the end-application
      is largely the same - it understands IP, DNS, and URIs.

      Imagine if Google required every client that connected with their
      service to participate in the proprietary Google query routing/messaging
      system. Of course, they chose to hide their service behind the
      abstraction of HTTP and their web services.

      In closing, JXTA has some great features, but end applications should be
      able to access JXTA services w/o knowing or caring that they are JXTA
      services.

      Peace,

      -Justin
    • Julian Bond
      ... A thought here. I ve been using Skype a lot recently. The one big USP about Skype is not the encryption, it s decentralised nature, the voice quality or
      Message 2 of 12 , Apr 23, 2004
        Justin Chapweske <justin@...> wrote:
        >Now, the last piece of this whole thing is HTTP proxying and caching.
        >Now, even after mentioning earlier how DNS/HTTP-based URIs may have
        >certain limitations, this all goes out the window when you consider HTTP
        >caching semantics.

        >If you need dynamic networking on a LAN, use zeroconf. If
        >you need dynamic networking among publicly addressable servers, use
        >DNS. If you need dynamic networking to NAT'd nodes, then use URIs and
        >HTTP. No matter which route you take, the code in the end-application
        >is largely the same - it understands IP, DNS, and URIs.

        A thought here. I've been using Skype a lot recently. The one big USP
        about Skype is not the encryption, it's decentralised nature, the voice
        quality or the cheap voice calls. It's that it *just works*. I've yet to
        come cross anyone who didn't just install it and have it work regardless
        of what firewall or NAT environment they're in. This is in marked
        contrast to every other voice/video system I've tried. What's remarkable
        about this is that they've done it without using a bank of high powered
        proxies with lots of bandwidth under their control or requiring
        complicated boundary proxies; which are the usual solutions. The problem
        of course is that the code is proprietary even if some of the techniques
        are fairly well understood now.

        Generalised IPv6 across the web feels just as remote as it did 3 years
        ago. There's more and more NAT everywhere as private users install WLAN
        routers on their broadband. The problem of public addressability isn't
        going away. It's getting worse.

        So to get to the point; Maybe there's scope here for using Skype/Kazaa
        like techniques but pushing them down the stack into the OS and App dev
        environment so that they fade from view. Let's make arbitrarily large
        numbers of directly connected machines into "Supernodes" or rather
        dynamic and temporary naming, routing and caching proxies. Somewhere in
        there is also an alternate name space that can't be controlled by
        Verizon, ICANN or any other gorilla. But that's another story.

        --
        Julian Bond Email&MSM: julian.bond at voidstar.com
        Webmaster: http://www.ecademy.com/
        Personal WebLog: http://www.voidstar.com/
        M: +44 (0)77 5907 2173 T: +44 (0)192 0412 433
      • Justin Chapweske
        ... True, but with Microsoft s recent SDKs, it looks like using IPv6 for a single application might be feasible since the whole thing appears to be tunnelable
        Message 3 of 12 , Apr 23, 2004
          > Generalised IPv6 across the web feels just as remote as it did 3 years
          > ago. There's more and more NAT everywhere as private users install WLAN
          > routers on their broadband. The problem of public addressability isn't
          > going away. It's getting worse.

          True, but with Microsoft's recent SDKs, it looks like using IPv6 for a
          single application might be feasible since the whole thing appears to be
          tunnelable over IPv4 UDP. Does anyone here have more direct experience
          with this?

          >
          > So to get to the point; Maybe there's scope here for using Skype/Kazaa
          > like techniques but pushing them down the stack into the OS and App dev
          > environment so that they fade from view.

          I agree. Application developers should be able to assume that all
          connections are point-to-point and bi-directional.

          I think the other camp of thinking is to explicitly assume that the
          network is heavily NAT'd and difficult to route in, and that application
          developers should explicitly deal with the problem via routing
          messages.

          For the time-being at least, I'm going to resist this mindset and push
          towards abstractions that allow transparent end-to-end communications.

          --
          Justin Chapweske - Founder, Onion Networks
          http://onionnetworks.com/
          651-340-8787

          Transfer large files 900% faster than FTP with Onion Networks' WAN
          Transport(tm). http://onionnetworks.com/products_wantransport.php
        • Wes Felter
          ... I mentioned this a while back, and people complained that Windows 9x, Mac OS X, and Linux don t support Teredo, so you have no hope of your app just
          Message 4 of 12 , Apr 23, 2004
            On Apr 23, 2004, at 9:25 AM, Justin Chapweske wrote:

            >> Generalised IPv6 across the web feels just as remote as it did 3 years
            >> ago. There's more and more NAT everywhere as private users install
            >> WLAN
            >> routers on their broadband. The problem of public addressability isn't
            >> going away. It's getting worse.
            >
            > True, but with Microsoft's recent SDKs, it looks like using IPv6 for a
            > single application might be feasible since the whole thing appears to
            > be
            > tunnelable over IPv4 UDP.

            I mentioned this a while back, and people complained that Windows 9x,
            Mac OS X, and Linux don't support Teredo, so you have no hope of your
            app "just working" on multiple platforms.

            Wes Felter - wesley@... - http://felter.org/wesley/
          • Justin Chapweske
            ... I have no doubt that those platforms would catch up pretty fast if a popular application was using Teredo. OS X and Linux do a pretty good job of
            Message 5 of 12 , Apr 23, 2004
              > >
              > > True, but with Microsoft's recent SDKs, it looks like using IPv6 for a
              > > single application might be feasible since the whole thing appears to
              > > be
              > > tunnelable over IPv4 UDP.
              >
              > I mentioned this a while back, and people complained that Windows 9x,
              > Mac OS X, and Linux don't support Teredo, so you have no hope of your
              > app "just working" on multiple platforms.

              I have no doubt that those platforms would catch up pretty fast if a
              popular application was using Teredo. OS X and Linux do a pretty good
              job of assimilating systems-level technologies like this.

              --
              Justin Chapweske - Founder, Onion Networks
              http://onionnetworks.com/
              651-340-8787

              Transfer large files 900% faster than FTP with Onion Networks' WAN
              Transport(tm). http://onionnetworks.com/products_wantransport.php
            • Eric Hopper
              ... Of course, my vision of the solution to this problem involves the creation of a location-transparent overlay network on top of IP. :-) But, even with such
              Message 6 of 12 , Apr 23, 2004
                On Thu, Apr 22, 2004 at 01:47:15PM -0500, Justin Chapweske wrote:
                > In order to provide dynamic networking at the URI naming level, you
                > simply need to switch a portion of your URIs to non-DNS-based URIs.
                > There is no need to switch all of your URIs to be location
                > independent, and indeed in Swarmcast we freely mix static HTTP URIs
                > with location independent UUID URIs.
                >
                > Location independent URIs, by their very nature, imply support for
                > both an anycast model, as well as a single-end-node or point-to-point
                > model. The only difference between these two cases is the number of
                > authoritative sources that can provide the service associated with the
                > URI in a secure fashion. In an anycast model, a number of nodes can
                > authoritatively serve a URI, and in a point-to-point model, only a
                > single end node can authoritatively serve a URI.

                Of course, my vision of the solution to this problem involves the
                creation of a location-transparent overlay network on top of IP. :-)

                But, even with such a thing, there still needs to be a way to
                dynamically map one network onto another, and I use URIs for that. In
                fact, since locations in my networking scheme can by a URI, this really
                is that same solution from a high-level perspective.

                Have fun (if at all possible),
                --
                "It does me no injury for my neighbor to say there are twenty gods or no God.
                It neither picks my pocket nor breaks my leg." --- Thomas Jefferson
                "Go to Heaven for the climate, Hell for the company." -- Mark Twain
                -- Eric Hopper (eric-yahoo@... http://www.omnifarious.org/~hopper) --
              • Eric Hopper
                ... Is there a pointer to something that describes what those techniques are? Thanks, -- It does me no injury for my neighbor to say there are twenty gods or
                Message 7 of 12 , Apr 23, 2004
                  On Fri, Apr 23, 2004 at 08:03:58AM +0100, Julian Bond wrote:
                  > complicated boundary proxies; which are the usual solutions. The
                  > problem of course is that the code is proprietary even if some of the
                  > techniques are fairly well understood now.

                  Is there a pointer to something that describes what those techniques
                  are?

                  Thanks,
                  --
                  "It does me no injury for my neighbor to say there are twenty gods or no God.
                  It neither picks my pocket nor breaks my leg." --- Thomas Jefferson
                  "Go to Heaven for the climate, Hell for the company." -- Mark Twain
                  -- Eric Hopper (hopper@... http://www.omnifarious.org/~hopper) --
                • Julian Bond
                  ... A quick google turned up http://www-rp.lip6.fr/teredo/ Teredo for BSD. So maybe the situation is improving. Another quick reading of the MS description
                  Message 8 of 12 , Apr 24, 2004
                    Wes Felter <wesley@...> wrote:
                    >I mentioned this a while back, and people complained that Windows 9x,
                    >Mac OS X, and Linux don't support Teredo, so you have no hope of your
                    >app "just working" on multiple platforms.

                    A quick google turned up http://www-rp.lip6.fr/teredo/ Teredo for BSD.
                    So maybe the situation is improving.

                    Another quick reading of the MS description pointed at IETF drafts such
                    as http://www.ietf.org/internet-drafts/draft-huitema-v6ops-teredo-01.txt

                    ISTM this still doesn't solve the discovery and initial connection
                    problem by itself. Teredo speaks of clients, servers and crucially,
                    relays. The Kazaa/Skype innovation was to decentralise the relays such
                    that any directly connected servent could potentially be a relay. In
                    Kazaa it's a conscious choice. In Skype, it's hidden and completely
                    automatic. I think Skype has also decentralised the naming service for
                    their namespace.

                    I haven't been able to find any independent technical analysis of Skype.
                    So you'll have to draw conclusions from their own page
                    http://www.skype.com/skype_p2pexplained.html

                    --
                    Julian Bond Email&MSM: julian.bond at voidstar.com
                    Webmaster: http://www.ecademy.com/
                    Personal WebLog: http://www.voidstar.com/
                    M: +44 (0)77 5907 2173 T: +44 (0)192 0412 433
                  • Paul Prescod
                    ... I can t believe that after all these years I am going to step back into the name versus location debate but I suppose I am just kind of nostalgic. Rather
                    Message 9 of 12 , Apr 24, 2004
                      > On Thu, Apr 22, 2004 at 01:47:15PM -0500, Justin Chapweske wrote:
                      >
                      >>In order to provide dynamic networking at the URI naming level, you
                      >>simply need to switch a portion of your URIs to non-DNS-based URIs.
                      >>There is no need to switch all of your URIs to be location
                      >>independent, and indeed in Swarmcast we freely mix static HTTP URIs
                      >>with location independent UUID URIs.

                      I can't believe that after all these years I am going to step back into
                      the name versus location debate but I suppose I am just kind of nostalgic.

                      Rather than write a long email I've written up a little essay. Please
                      excuse the fact that it is barely edited:

                      http://www.prescod.net/rest/combining_names_and_locations/

                      >>Location independent URIs, by their very nature, imply support for
                      >>both an anycast model, as well as a single-end-node or point-to-point
                      >>model. The only difference between these two cases is the number of
                      >>authoritative sources that can provide the service associated with the
                      >>URI in a secure fashion. In an anycast model, a number of nodes can
                      >>authoritatively serve a URI, and in a point-to-point model, only a
                      >>single end node can authoritatively serve a URI.

                      These features all hold regardless of whether the URI has a syntax
                      starting with "http://" or "uuid:" You can choose to use an anycast
                      model for HTTP URIs, you can have multiple authoritiative sources for
                      HTTP URIs (haven't you ever used Google's cache as more authoritiative
                      than the referenced domain?) etc.

                      Paul Prescod
                    • Peter Ferne
                      ... There is port of Teredo for Free BSD (http://www-rp.lip6.fr/teredo/) so I guess getting it working on OS X at least shouldn t be *too* much work... --
                      Message 10 of 12 , Apr 25, 2004
                        > I mentioned this a while back, and people complained that Windows 9x,
                        > Mac OS X, and Linux don't support Teredo, so you have no hope of your
                        > app "just working" on multiple platforms.

                        There is port of Teredo for Free BSD (http://www-rp.lip6.fr/teredo/) so
                        I guess getting it working on OS X at least shouldn't be *too* much
                        work...
                        --
                        Peter Ferne, +44 (0)7970 942 261, properdigital@...
                      • Lucas Gonze
                        On Saturday, Apr 24, 2004, at 21:52 America/New_York, Paul Prescod ... For a name that isn t an address, what advantage does a name that starts with http://
                        Message 11 of 12 , Apr 25, 2004
                          On Saturday, Apr 24, 2004, at 21:52 America/New_York, Paul Prescod
                          wrote:
                          > These features all hold regardless of whether the URI has a syntax
                          > starting with "http://" or "uuid:" You can choose to use an anycast
                          > model for HTTP URIs, you can have multiple authoritiative sources for
                          > HTTP URIs (haven't you ever used Google's cache as more authoritiative
                          > than the referenced domain?) etc.

                          For a name that isn't an address, what advantage does a name that
                          starts with http:// have?
                        • Paul Prescod
                          ... 1. The ability to _become_ an address months or years after the name has been deployed. For instance, XML namespaces have started to sprout RDDL documents
                          Message 12 of 12 , Apr 25, 2004
                            Lucas Gonze wrote:

                            > On Saturday, Apr 24, 2004, at 21:52 America/New_York, Paul Prescod
                            > wrote:
                            >
                            >>These features all hold regardless of whether the URI has a syntax
                            >>starting with "http://" or "uuid:" You can choose to use an anycast
                            >>model for HTTP URIs, you can have multiple authoritiative sources for
                            >>HTTP URIs (haven't you ever used Google's cache as more authoritiative
                            >>than the referenced domain?) etc.
                            >
                            >
                            > For a name that isn't an address, what advantage does a name that
                            > starts with http:// have?

                            1. The ability to _become_ an address months or years after the name has
                            been deployed. For instance, XML namespaces have started to sprout RDDL
                            documents years after they were first let out into the wild.

                            2. The ability to point at its own documentation: "The URI you have
                            dereferenced does not have any machine readable content but you may be
                            curious about what that URI means. Here's what I know about it." This is
                            also very common in the XML world: where there isn't a RDDL document at
                            the end of a URI, there is often an HTML document.

                            I know I've plugged UUIDs into Google to figure out what they mean, but
                            that's a hit or miss proposition. Dereferencing HTTP URIs is much more
                            reliable. Having the opportunity to do both is ideal. e.g. you can
                            either dereference the HTML 4 namespace URI or Google it.

                            [1] http://www.w3.org/TR/html4/

                            [2]
                            http://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&q=%22%2Bwww.w3.org/TR/html4/%22

                            Paul Prescod
                          Your message has been successfully submitted and would be delivered to recipients shortly.