>  However, you must admit that we have a somewhat serious
>  educational problem here. The MLDonkey developer doesn't understand
>  the exponential cost associated with broadcast queries [of any type].

There is no "educational problem". I've been a teacher for a while,
and I was happy to see when of my students discussing what was in the
lecture, as soon as he is not completely stubborn. I was just
complaining on the fact that we cannot guess these values (why one
hour and not two :) ), and suggesting to add a new header in handshakes:
Implement-Standard-Gnutella-Client-Behavior: 0.1 :), to test clients
complying to a proposal containing these informations.

Anyway, these delays will be added in MLdonkey as soon as possible.

>  What do you propose we do as a solution going forward, so that this
>  mistake is not repeated ad nauseum?

Why not restrict the number of queries sent by each leaf at the
ultrapeer level ? It could be easy to cache the queries, and only
propagate the ones that have not yet been sent in the last hour (don't
disconnect the leaves, just degrade the service).

In Edonkey, all servers now implement some control on leaves, such as
limiting the number of shared files, and the number of queries per
minute.

- b8_bavard (mldonkey)
--------------------------------------
Homepage: http://www.mldonkey.net/
--------------------------------------

--- In the_gdf@yahoogroups.com, b8_bavard <mldonkey@m...> wrote:
> There is no "educational problem".

As Abaris mentioned, the problem is a lack of documentation.

>> Why not restrict the number of queries sent by each leaf at the
>> ultrapeer level ? It could be easy to cache the queries, and only
>> propagate the ones that have not yet been sent in the last hour (don't
>> disconnect the leaves, just degrade the service).

What about doing the following on leaf connections? It's just a wild
idea, so feel free to criticism.

(1) Every automated requery must contain a GGEP marker extension "AR".
     Any Query not carrying this extension is considered human generated.

(2) Associate a simple timestamp with each connection holding the last
     time an "AR"ed query was seen. Watch for leaves that disobey the
     requery limit of 1 requery per hour total.

(3) Use a more generous limit based on sensible heuristics for human
     generated (non-"AR"ed) queries, with respect to the fact that users
     usually type in bursts, but are then silent for a longer period of time.
     The average number of queries sent by decent servants and the maximum
     number of queries an Ultrapeer can handle should also be taken into
     account. This is the most important and complicated part, but I lack
     the necessary statistics to go further into this. NO user will ever
     query the network once every 12 seconds for several minutes (like
     XoloX 1.4.1 did).

(4) If one of these limits is not obeyed, take countermeasures with respect
     to the order of damage:

         (a) Drop additional queries. Drop "AR"ed queries before regular ones.

         (b) Throttle down the receive bandwidth donated to this leaf so it
             can't seriously damage the Ultrapeer's funcionality. If it still
             manages to pass the limit, drop additional queries as in (a).

         (c) Disconnect the leaf.

An excessive requeryer should experience a serious decrease of functionality
once this System gets deployed. It's users should urge the developer to update.
The "AR" marker seems unnecessary on first glance, and perhaps it is. However,
there is at least one good reason to include it:

Automated requeries sent by conforming clients should have a lower
priority than human generated queries. This gets important on the
intra-Ultrapeer level, where the above rules do not apply. Excessive
messaging is typically handled by flow control, where the first
messages to drop should be automated requeries. This helps making
sure that requerying will not decrease the effect of those querys
that are supposed to return results in the first place.

Furthermore, servants that are willing to conform but are just buggy
(i.e., those who include "AR" but pass the limit anyways) can be handled
in a way so that requerys are dropped but other queries are not.

This system doesn't help to defend the network against malicious servants
that are going to not put the marker in its requeries, masquerading requeries
as regular queries. However, the above heuristics' result will be that many
of these servant's regular queries will be dropped along with the requeries,
seriously affecting end user experience compared to the servant above
who also passes the limit, but does include the marker.

  - thanks


[Non-text portions of this message have been removed]

From: "Vinnie" <vinnie@...>
> --- In the_gdf@yahoogroups.com, b8_bavard <mldonkey@m...> wrote:
> > There is no "educational problem".
>
> As Abaris mentioned, the problem is a lack of documentation.

Most important is a lack of a document that gives the good hints for developers,
so that the concept of "broadcast" routing in Gnutella (for queries and pings)
be severely controled by mathematical considerations that consider the network
topology and the influence of critical parameters (notably the TTL and Hops)

We should be able to design a dynamic processing model for "broadcasts" by
documenting how these "broadcasts" can be handled more dynamically: ie. favoring
their routing using non classical and static 1-to-N delivery, but dynamic 1-to-1
delivery where each connection is tried in a delayed sequence.

This requires defining a dynamic context for all incoming queries and pings,
where the old "TTL" concept is replaced by more dynamic metrics. We should
obsolete the old rules for these TTL static fields, by saying "TTL values
considered harmful". We should then be able to specify for each query to unknown
destinations their expected metrics for returns, and their expected propagation
for the search itself, and nodes should be able to indicate to their peers their
capabilities for searches in terms of these metrics (maximum acceptable query
rates, and expected maximum return rate), so that the network would learn how to
browse and best use available resources...

The main reason Gnutella is abused is really the fact that routing rules for
broadcasts has not been documented in terms of metrics, and with mathematic
considerations and result reports. So it's true that we need some form of
broadcast to perform some searches, but each time a broadcast is needed,
developers should ask themselves what will be the impact on the network, and
specify limiting factors.

May be we could think about defining a new Query message with a much more
precise routing specification, which would also add semantics to search, support
internationalization and advanced filters, and would also transport dynamic
metrics about its past and expected propagation.
The message would have two distinct parts:
- one which can be signed contains the query parameters and filters, and
extensible identity info about the querier (including information for direct
routing of the answer back to the querier).
- another one which is modifiable at each hop, and controls its propagation, can
trace the route already followed by the query, allow notifying back a past
proxying agent when a critical condition occurs (such as a metric that exceeds
some dynamic thresholds, or can say that the propagation has been terminated at
some point). This part would allow computing all statistical metrics, and take
into consideration the locally discovered topology (not known by the querier
which may be far away in terms of hops, or even "disconnected" in case of
transmission through UDP or fallback crawling TCP connections).

As the content of such message would not be clear, I think this message should
have an open content-model with multiple encapsulation levels, each one allowed
to include signature data if needed, much like an XML document (in terms of DOM
nodes, not necessarily in terms of syntax, which could be encoded with binary
TLV structures like in Mac resources or ILBM images).

What seems important when defining such content model is to define at each level
of the DOM an experimental area where a vendor experiments contents specific to
a set of versions of its servent, always as an extension (a sub-schema) of a
more general model initially empty. Each level in the DOM tree would be assigned
to a unique entity that documents and implements it as a reference model.
Separate specifications could never be defined within a node of a specific
vendor, but each endor should include a space in its DOM node for extensions or
separate experimentations.

Efficiency would be important; we don't need to se the text/xml format which
reqires implementing a complex parser. XML is defined in terms of DOM, not by
its syntax (text/xml is just a reference implementation that should comply to
the DOM specification, and some revisions to the text/xml documentation is
sometimes done only to comply with XML-DOM requirements.)
What is important in XML is not the fact that elements can be named, but the
existence of namespaces, allowing elements to be renamed without colliding each
other.

This allows us to restrict what can be node names, and we can defined them in a
restricted way to avoid using dictionnary based parsers and the maintenance of
compressed dictionnaries (which is a real problem when converting a generic XML
document to binary format), in favor of fixed-size symbols (such as 8, 16 or
32-bit ints, like in MIBs) allocated specifically for each node family by the
vendor defining it. In my opinion, the original general purpose subfields should
be given small IDs, and all extensions or private fields should have an ID
allocated to a vendor that requests it to add a sub-schema (for experimentation
or private use).

The other argument in favor of such XML-DOM like encapsulation is the capability
of creating signature containers: such containers for a specific node could be
predefined so that it can be used anywhere a node can be interted in the DOM
tree, and this must not prevent a peer to interpret standard fields.

If we used a binary representation, this would be something like transforming
the generic {T,L,V} format (Type, Length, Value) of a node in the DOM to
{T,L,{n,V,s}} where L is the total length of {n,V,s}, V is the contained node, n
is an optional length specifier for the signature data "s" (the signature itself
is not prefixed by a specific type, but it could be also represented by a
{t,n,s} triplet if needed, so that the final representation would be
{T,L,{n,V,{t,s}}}, once the redundant field n is removed (so the contained data
has length L-n, found in a fixed position within the binary node representation.

Another better representation to meet the goal of allowing the interoperability
with nodes without signature could be simply to insert a control flag in the
header of the TLV node to specify that the node is followed by another node
which is the signature data for the previous node. Nodes would then have a
category: isolated data (not followed by signature), secured data (followed by a
signature), signature, and possibly another alternate category for encryption
challenge key (when the data value is opaque and not usable without decrypting
it with the key, however its node type would not be encrypted). This requires
just 2 bits in the binary DOM node header, or 1 bit set to 0 for data nodes
(taken in the "T" type field of the data node, and 3 bits for special nodes
(leading bit set to 1, two other control bits being standardized, and taken in
the "t" type field of the special node).

Then we need to specify how to encode efficiently TLV nodes of varying length,
without wasting space, given that the dictionnary of types (T) (equivalent to
XML-DOM node names) must still be opened to extensions and versioning. I won't
militate in favor of requiring a version field in each node. For me each version
would be a separate type, and each level in the DOM tree schema defines its own
dictionnary, which has its own rules for allocating subnode type values (this
allows a compact representation at each level in the tree, and possible
simplified remapping to new subnode types in another version of the defining
node type if categorizing is necessary).

Node type allocations and semantics could then be formally defined by an open
DTD or XML-schema, which lists subtypes allong with a href link to a document
describing it for implementors, and links to the schemas for sub node types in
the DOM tree). Using such separate but formal strict DTD or XMLS schema
definition could allow servents to build an internal and compact database of
node types supported along with their properties (notably if the value is a
string and can be indexed, or is a anchor or reference in the data, or allows to
define compatibility equivalences with other newer types which extends an
original version). This open shema would work like a multivendor repository,
where could be stored semantics (as reference documents, in TXT or HTML orPDF
format)and encoding rules for the binary representation (an alternate text/xml
representation could be deduced from it).

This work done, we could migrate progressively existing messages to the new
message formats, and it would no more be needed to use GGEP or GEM or other
legacy extension formats. The new system being defined by its DOM structure,
there could also be alternate binary representations if needed in the future
(without needing to recreate the DOM schemas), that would allow easier
interoperability with other P2P networks (such as the emerging JAXP, SOAP,
WebXML, or .net services) using more efficient encodings: just think about
Unicode, there was a time where it was encoded with UCS-2 (as a simplified way
to encode ISO10646), then UTF-7, and UTF-8, then UTF-16 deprecating UCS-2, now
BOCU and SCSU... Think about compression algorithms, media formats...

In the future, XML parsers will be so common and so efficients, that even the
"text/xml" lingua-franca will be generally usable along with standard
compression algorithms, to be the best binary format for interoperability and
efficiency (by the dramatic reduction of the code size and development time
needed to support it in all applications), because all applications will speak
natively in terms of XML-DOM interfaces and won't bother to know its internal
encoding.

If we want Gnutella to remain an open-standard, we must still allow it to evolve
(slowly) in the same direction as emerging standards that will provide better
services.

The fondamental need for a peer-to-peer network is to be able to query it
efficiently without knowing its exact topology. This requires much more dynamic
analysis of the traffic and more works in innovative routing systems that can
use partial local knowledge of capabilities of adjascent or accessible nodes in
the network. For now Gnutella has only survived by enforcing some static metrics
but in the long term, this strategy is not viable, and the Gnutella fondamentals
must evolve (notably because it depends too much on the capability of some nodes
to accept incoming direct connections and support most of the traffic, and this
is a major problem in terms of scalability as too many -- leaf -- nodes can't be
used nominaly as they should).

Hello,

I have noticed the hostiles.txt file which is used by bearshare
clients is also available using their bearhshare its website. Is it ok
for a client to automatically fetch this file (say once a week?)
automatically. Or should this be avoided?

- Jeroen

On Wednesday 23 April 2003 03:11 pm, Vinnie wrote:
>
> Testing against LimeWire is even MORE useful, since it is open
> source. The existence of Open Source LimeWire eliminates any
> possibility of excuses for not debugging a new servent, and Java runs
> on ALL PLATFORMS.

It's not quite that simple, you have to search for and disable the network
read timeouts that tend to break off communications while you are stepping
through the logic.   It can be done, but I've often found it easier and just
as informative to add print statements to dump the stack trace when Limewire
rejects a connection.

Hi,
I know this is not a very technical question, but I find it really interesting
that Bearshare Ultrapeers almost never drop compared to LimeWire or even
Gnucleus. I think the answer to this could help a lot to improve the Gnutella
network.

Thanks in advance
Jann

--- In the_gdf@yahoogroups.com, "Jeroen Asselman" <nonamer_nl@y...>
wrote:
> I have noticed the hostiles.txt file which is used by bearshare
> clients is also available using their bearhshare its website. Is it
ok
> for a client to automatically fetch this file (say once a week?)
> automatically. Or should this be avoided?

This should be avoided. If we detect any automated program accessing
that file it will be removed promptly.

Quoting Adam Fisk <afisk@...> from ml.gnutella.dev-forum:
:We're making changes for our next release that will make dynamic
:querying far more conservative.  We're now waiting between iterations
:about 2200 milliseconds per hop of the previous query, which matches the
:latencies we've seen on the network.  For example, if we just sent a
:query at TTL=3 down one connection, we'll wait 6600 milliseconds for
:results to come in before calculating the next TTL and sending out the
:next query.  This increases the overall latency of the query
:significantly, but saves a heck of a lot of bandwidth.  These changes
:should quiet the network down even more, hopefully allowing us to be
:even more conservative in the future.

Still waiting for your specifications to implement that in GTKG, along
with the high outdegree.

If you publish them this week, they will be in the final 0.92 release
planned for the end of the month.  If you miss that target, they will have
to appear in 0.93, planned for next year only.

When you do publish your specs, please be sure to document those tuning
parameters and guidelines, so that everyone does not need to reinvent
the wheel.

Thanks.

Raphael

Vinnie wrote:

> --- In the_gdf@yahoogroups.com, "Jeroen Asselman" <nonamer_nl@y...>
> wrote:
>
>>I have noticed the hostiles.txt file which is used by bearshare
>>clients is also available using their bearhshare its website. Is it ok
>>for a client to automatically fetch this file (say once a week?)
>>automatically. Or should this be avoided?
>
> This should be avoided. If we detect any automated program accessing
> that file it will be removed promptly.

Why do you make it public then?

--- Vinnie <vinnie@...> wrote:
> --- In the_gdf@yahoogroups.com, "Jeroen Asselman"
> <nonamer_nl@y...>
> wrote:
> > I have noticed the hostiles.txt file which is used
> by bearshare
> > clients is also available using their bearhshare
> its website. Is it
> ok
> > for a client to automatically fetch this file (say
> once a week?)
> > automatically. Or should this be avoided?
>
> This should be avoided. If we detect any automated
> program accessing
> that file it will be removed promptly.

OK,

I'll have to find another safe way for keeping this
file up to date than.

=====
-- Jeroen

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

--- In the_gdf@yahoogroups.com, Jeroen Asselman <nonamer_nl@y...>
wrote:
>
> --- Vinnie <vinnie@f...> wrote:
> > --- In the_gdf@yahoogroups.com, "Jeroen Asselman"
> > <nonamer_nl@y...>
> > wrote:
> > > I have noticed the hostiles.txt file which is used
> > by bearshare
> > > clients is also available using their bearhshare
> > its website. Is it
> > ok
> > > for a client to automatically fetch this file (say
> > once a week?)
> > > automatically. Or should this be avoided?
> >
> > This should be avoided. If we detect any automated
> > program accessing
> > that file it will be removed promptly.
>
> OK,
>
> I'll have to find another safe way for keeping this
> file up to date than.
>
> =====
> -- Jeroen
>

May I suggest Jeroen download this file himself once a week (one
little download won't hurt) and then host a copy on his own website
for his clients to access automatically. He should even make efforts
to detect other hostiles, add them to his copy and then maybe inform
bearshare of his findings so they can choose whether to add them to
their list.
Am I right in thinking there is currently no cooperation in this
manner with other vendors?

> When you do publish your specs, please be sure to document those tuning
> parameters and guidelines, so that everyone does not need to reinvent
> the wheel.

I would be very interested in these specs too...

Gregor

--
+++ GMX - Mail, Messaging & more  http://www.gmx.net +++
Bitte lächeln! Fotogalerie online mit GMX ohne eigene Homepage!

We find that LimeWire's stay up the longest.  Perhaps you just have a richer
mix of BearShare pongs that leads you to stay connected to BearShare hosts.

Thanks
-greg
   -----Original Message-----
   From: jann.roeder@... [mailto:jann.roeder@...]
   Sent: Sunday, May 04, 2003 4:05 PM
   To: The GDF
   Subject: [the_gdf] Why are Bearshare Ultrapeers so stable compared to
others ?


   Hi,
   I know this is not a very technical question, but I find it really
interesting that Bearshare Ultrapeers almost never drop compared to LimeWire
or even Gnucleus. I think the answer to this could help a lot to improve the
Gnutella network.

   Thanks in advance
   Jann


         Yahoo! Groups Sponsor



   To unsubscribe from this group, send an email to:
   the_gdf-unsubscribe@egroups.com



   Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.



[Non-text portions of this message have been removed]

> May I suggest Jeroen download this file himself once
> a week (one
> little download won't hurt) and then host a copy on
> his own website
> for his clients to access automatically. He should
> even make efforts
> to detect other hostiles, add them to his copy and
> then maybe inform
> bearshare of his findings so they can choose whether
> to add them to
> their list.
> Am I right in thinking there is currently no
> cooperation in this
> manner with other vendors?

I wonder if it would be possible for bearshare to sign
this file so it can be updated using gnutella. All we
would need is a public key. (Or wouldn't this work?)

- Jeroen

=====
-- Jeroen

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

IANAL!!!!

Perhaps legal liability issues prevent an update-on-demand blocking-by-IP
feature?

I am sure that there would be legal ramifications, in any case, that would have
to be considered.

        -dave-



-----Original Message-----
From: Jeroen Asselman [mailto:nonamer_nl@...]
Sent: Monday, May 05, 2003 11:37 AM
To: the_gdf@yahoogroups.com
Subject: Re: [the_gdf] Re: regarding bearshare' hostiles.txt


> May I suggest Jeroen download this file himself once
> a week (one
> little download won't hurt) and then host a copy on
> his own website
> for his clients to access automatically. He should
> even make efforts
> to detect other hostiles, add them to his copy and
> then maybe inform
> bearshare of his findings so they can choose whether
> to add them to
> their list.
> Am I right in thinking there is currently no
> cooperation in this
> manner with other vendors?

I wonder if it would be possible for bearshare to sign
this file so it can be updated using gnutella. All we
would need is a public key. (Or wouldn't this work?)

- Jeroen

=====
-- Jeroen

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Yahoo! Groups Sponsor





To unsubscribe from this group, send an email to:
the_gdf-unsubscribe@egroups.com



Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.

Well I don't know much about legal issues either but I would find it very
usefull to have a place where trusted vendors could maintain this hostile list.
I have no problem to deliver a updated list with every release, if it would
be a problem to have it weekly automaticaly updated by the client.
But the collection of the hostile IPs should be something every vendor can
participate in. For Phex I also started using this list and also did my own
extension of it. Would be nice to be able to share and regularly update the
list from other vendor.

Gregor


> IANAL!!!!
>
> Perhaps legal liability issues prevent an update-on-demand blocking-by-IP
> feature?
>
> I am sure that there would be legal ramifications, in any case, that would
> have to be considered.
>
>        -dave-
>
> -----Original Message-----
> From: Jeroen Asselman [mailto:nonamer_nl@...]
> Sent: Monday, May 05, 2003 11:37 AM
> To: the_gdf@yahoogroups.com
> Subject: Re: [the_gdf] Re: regarding bearshare' hostiles.txt
>
>
> I wonder if it would be possible for bearshare to sign
> this file so it can be updated using gnutella. All we
> would need is a public key. (Or wouldn't this work?)
>
> - Jeroen

--
+++ GMX - Mail, Messaging & more  http://www.gmx.net +++
Bitte lächeln! Fotogalerie online mit GMX ohne eigene Homepage!

Quoting Greg Bildson <gbildson@...> from ml.gnutella.dev-forum:
:We find that LimeWire's stay up the longest.  Perhaps you just have a richer
:mix of BearShare pongs that leads you to stay connected to BearShare hosts.

I found that GTKG's stay up the longest, then Shareaza/BearShare, then
Morpheus/Gnucleus.

LimeWire ultrapeers usually give me the worst possible experience:
they disconnect abruptly, and since they don't send any BYE message, it's
hard to tell what went on.

I'm currently trapped in a BearShare cloud of UPs...

Raphael

Hmmm.  Given how strongly connected the LimeWire ultrapeers tend to get
right now, it is probably only the very transient LimeWires that are seen by
other venders.  We are working to correct this.

Thanks
-greg
   -----Original Message-----
   From: Raphael_Manfredi@... [mailto:Raphael_Manfredi@...]
   Sent: Monday, May 05, 2003 1:03 PM
   To: the_gdf@yahoogroups.com
   Subject: [the_gdf] Re: Why are Bearshare Ultrapeers so stable compared to
others ?


   Quoting Greg Bildson <gbildson@...> from ml.gnutella.dev-forum:
   :We find that LimeWire's stay up the longest.  Perhaps you just have a
richer
   :mix of BearShare pongs that leads you to stay connected to BearShare
hosts.

   I found that GTKG's stay up the longest, then Shareaza/BearShare, then
   Morpheus/Gnucleus.

   LimeWire ultrapeers usually give me the worst possible experience:
   they disconnect abruptly, and since they don't send any BYE message, it's
   hard to tell what went on.

   I'm currently trapped in a BearShare cloud of UPs...

   Raphael

         Yahoo! Groups Sponsor
               ADVERTISEMENT




   To unsubscribe from this group, send an email to:
   the_gdf-unsubscribe@egroups.com



   Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.



[Non-text portions of this message have been removed]

I'll start on these specs tonight.  Again, much of it will be more
general, particularly with regard to precisely what parameters to use.
I'd be more than happy to tell everyone precisely what we're doing here,
but it might not make sense to do that for the spec.  For example, we
pause for about 2200 milliseconds per hop per connections used, as I
alluded to earlier (we'll wait 6600 milliseconds after sending a query
with TTL=3 down 1 connection).  If it's really clear that a file is very
rare, though, we may gradually lessen that time.  The logic here would
be that if it's clear we're going to query as many people as we
realistically can, you might as well do it more quickly.  Decisions like
these will probably be outside the scope of the spec, though, as they're
likely to change.  There are also undoubtedly a lot of good ideas out
there that would make an overly specific spec obsolete pretty quickly.

So, the specs will more outline these ideas as the general direction we
clearly want to go in, but it will leave lots of room for innovative
ideas to fill in the blanks.

-adam


Gregor Koukkoullis wrote:

>>When you do publish your specs, please be sure to document those tuning
>>parameters and guidelines, so that everyone does not need to reinvent
>>the wheel.
>>
>>
>
>I would be very interested in these specs too...
>
>Gregor
>
>
>

Quoting Abaris <abaris23@...> from ml.gnutella.dev-forum:
:What about doing the following on leaf connections? It's just a wild
:idea, so feel free to criticism.
:
:(1) Every automated requery must contain a GGEP marker extension "AR".
:    Any Query not carrying this extension is considered human generated.

It is a good idea to tag automated requeries.

However, it can only be useful for servents that accept to play ball.
A malicious servent would not abide by those specs.

Note that if your purpose is to detect requerying at the UP level for
the leaves, then this is fairly easy to do...  It's for the relayed traffic
that you indeed need a marker.

FYI, GTKG tags its queries and its requeries in the GUID.

It tag its queries to be able to respond with the GGEP "H" extension for
the SHA1, instead of the urn:sha1:XXXX in ASCII.  Apparently, only GTKG saw
interest in the "H" extension...

As for tagging requeries, this is for statistics purposes only.

Raphael

Quoting Adam Fisk <afisk@...> from ml.gnutella.dev-forum:
:I'll start on these specs tonight.

Thanks Adam.

If you can have a draft ready by the Wednesday, I'll be able to give you
feedback on the specs whilst I implement it, since I'll be working full
time on GTKG starting Thursday morning until Sunday evening this week...

Raphael

Quoting Greg Bildson <gbildson@...> from ml.gnutella.dev-forum:
:Hmmm.  Given how strongly connected the LimeWire ultrapeers tend to get
:right now, it is probably only the very transient LimeWires that are seen by
:other venders.  We are working to correct this.

It would help if, when purposedly disconnecting, servents were sending a
BYE message.  It does not cost much, and it can help diagnose a possible
problem.

Without any feedback, I cannot tell why the connection is dropped.  All I
can tell is that it is not GTKG that drops it.

Raphael

Quoting Philippe Verdy <verdy_p@...> from ml.gnutella.dev-forum:
:The 0.56 specs is the version of the last "official" document for the 0.4
:protocol edited by Clip2 before it got out of business.
:However, this spec was not extremely clear in some features, and so many
:implementers have expected that all strings be null terminated, and use a
common
:format to encode both queries and query hits.

Queries have always been single-NUL terminated.

Therefore, only ONE NUL is necessary to delimit the end of the query
string.  Anything after that is just extension, and therefore the extension
part SHOULD NOT be NUL-terminated.

There is little room for arguing here.  Those are just the specs.

Raphael

--- In the_gdf@yahoogroups.com, Jann Röder <jann.roeder@w...> wrote:
> Bearshare Ultrapeers almost never drop

Hold on, first there are complaints that BearShare Ultrapeers are
impossible to connect to, now we hear that they have high uptimes?

Which one is it?

--- In the_gdf@yahoogroups.com, Raphael_Manfredi@p... wrote:
> It would help if, when purposedly disconnecting, servents were
sending a
> BYE message.  It does not cost much, and it can help diagnose a
possible
> problem.
>
> Without any feedback, I cannot tell why the connection is dropped.
All I
> can tell is that it is not GTKG that drops it.

That the current version of BearShare *NEVER* drops connections on
purpose. The only way a connection is dropped is under the following
circumstances (understandably):

1) There is a network connectivity issue (a cable is unplugged, a key
router goes down, etc..)

2) The user manually removes a host from the list (extremely rare,
since the Hosts View is hidden by default)

3) The application is terminated manually, terminated due to a
defect, or power is removed from the machine

4) The message stream becomes desynchronized (can happen when
receiving corrupt data)

Let me repeat that the ONLY code path where a host connection will be
dropped (in the non manual case) is #4 above, and this condition is
reported in the console. I have personally observed corrupted host
message streams an extremely small number of times, and mostly due to
a bug we had a while back when BearShare 4.0.0 was first released.

Therefore, while I admire Raphael's interest in a verbose description
of WHY the connection was purposely closed, it should be very obvious
that when it comes to BearShare at least, only the first three of the
reasons above are applicable, and a BYE message would never make it
to the other end in those cases anyway.

We have intentionally designed BearShare so there are no code paths
that lead to host connections being dropped automatically, except for
the case where the message stream is desynchronized (and the obvious
case of exiting).

The conclusion is that BearShare servents don't drop connections
willy-nilly; Connections are mainly dropped because the user exits
the application.

LimeWire also will not intentionally drop a connection.

Thanks
-greg
   -----Original Message-----
   From: Vinnie [mailto:vinnie@...]
   Sent: Monday, May 05, 2003 3:41 PM
   To: the_gdf@yahoogroups.com
   Subject: [the_gdf] Re: Why are Bearshare Ultrapeers so stable compared to
others ?


   --- In the_gdf@yahoogroups.com, Raphael_Manfredi@p... wrote:
   > It would help if, when purposedly disconnecting, servents were
   sending a
   > BYE message.  It does not cost much, and it can help diagnose a
   possible
   > problem.
   >
   > Without any feedback, I cannot tell why the connection is dropped.
   All I
   > can tell is that it is not GTKG that drops it.

   That the current version of BearShare *NEVER* drops connections on
   purpose. The only way a connection is dropped is under the following
   circumstances (understandably):

   1) There is a network connectivity issue (a cable is unplugged, a key
   router goes down, etc..)

   2) The user manually removes a host from the list (extremely rare,
   since the Hosts View is hidden by default)

   3) The application is terminated manually, terminated due to a
   defect, or power is removed from the machine

   4) The message stream becomes desynchronized (can happen when
   receiving corrupt data)

   Let me repeat that the ONLY code path where a host connection will be
   dropped (in the non manual case) is #4 above, and this condition is
   reported in the console. I have personally observed corrupted host
   message streams an extremely small number of times, and mostly due to
   a bug we had a while back when BearShare 4.0.0 was first released.

   Therefore, while I admire Raphael's interest in a verbose description
   of WHY the connection was purposely closed, it should be very obvious
   that when it comes to BearShare at least, only the first three of the
   reasons above are applicable, and a BYE message would never make it
   to the other end in those cases anyway.

   We have intentionally designed BearShare so there are no code paths
   that lead to host connections being dropped automatically, except for
   the case where the message stream is desynchronized (and the obvious
   case of exiting).

   The conclusion is that BearShare servents don't drop connections
   willy-nilly; Connections are mainly dropped because the user exits
   the application.



         Yahoo! Groups Sponsor





   To unsubscribe from this group, send an email to:
   the_gdf-unsubscribe@egroups.com



   Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.



[Non-text portions of this message have been removed]

---------- Original Message ----------------------------------
From: Raphael_Manfredi@...
>Quoting Abaris <abaris23@...> from ml.gnutella.dev-forum:
>:What about doing the following on leaf connections? It's just a wild
>:idea, so feel free to criticism.
>:
>:(1) Every automated requery must contain a GGEP marker extension "AR".
>:    Any Query not carrying this extension is considered human generated.
>
>It is a good idea to tag automated requeries.
>
>However, it can only be useful for servents that accept to play ball.
>A malicious servent would not abide by those specs.
>
>Note that if your purpose is to detect requerying at the UP level for
>the leaves, then this is fairly easy to do...  It's for the relayed traffic
>that you indeed need a marker.
>
>FYI, GTKG tags its queries and its requeries in the GUID.
>
>It tag its queries to be able to respond with the GGEP "H" extension for
>the SHA1, instead of the urn:sha1:XXXX in ASCII.  Apparently, only GTKG saw
>interest in the "H" extension...
>
>As for tagging requeries, this is for statistics purposes only.
>
>Raphael

LimeWire encoded 3 different generations of requeries in our GUIDs.  As we moved
to new versions, we killed off all the queries of the older versions that we saw
because they were too aggressive.  We learned that requeries were very tricky
and very high bandwidth.

Thanks
-greg

Vinnie wrote:
> --- In the_gdf@yahoogroups.com, Jann Röder <jann.roeder@w...> wrote:
>
>>Bearshare Ultrapeers almost never drop
>
>
> Hold on, first there are complaints that BearShare Ultrapeers are
> impossible to connect to, now we hear that they have high uptimes?
>
> Which one is it?

Both. In fact, one is a direct result of the other.

You have stated that Bearshare will never drop an 'outsider' in favor of
a Bearshare node if it is taking one of those 5 slots.

<quote name=Vinnie>
	 I can see how our statements might seem at odds. With 4.2.5
	 reserving 5 slots for non-BS leaves, it is forcing BearShare to
	 accept non-BS leaves, and those 5 slots are available to any
	 vendor. None of those 5 slots are ever dropped in favor of a
	 Bear.
</quote>

However, you do not state that those 5 slots are reserved SPECIFICALLY
for non-Bearshare clients. I doubt you have code that would kick a
Bearshare client off in favor of another vendor.


So, let's take a look at the life of a Bearshare node...

A Bearshare node promotes itself to an Ultrapeer, and starts accepting
leaves. 100 clients request connections, 15 are non-Bearshare... 5
non-Bearshare clients get a connection with you.

Now, over time, some connections will be dropped, and will be replaced
in a First-Come-First-Serve manner. Due to the fact that 85% of your
incoming requests are Bearshare leaves, those '5 reserved slots' will
rather quickly become 5 Bearshare slots... you may not KICK a
non-Bearshare client out of them, but you definetly won't kick a
BEARSHARE client out of a reserved slot in favor of an outsider.

The median leaf lifetime is about an hour, right? So in an hour or two,
your '5 reserved slots' are '5 bearshare slots' again. The window of
opportunity for outsider clients is pretty slim.

To truly make it feasable for outsider clients to get a connection, you
would kick Bearshare clients out of the 5 reserved slots when an
outsider attempted to connect. That is, of course, not in your best
interests... though it WOULD be in the best interests of the network,
allowing new servants to at least make the attempt to test their servant.

So this is why Bearshare can have high uptime AND make it difficult for
other vendors to connect to. High uptime Bearshare nodes, with all
Bearshare leaves, that never die wiping the slate clean. Few new
Bearshare Ultrapeers come on at any particular period of time because
they stay online so long... few new nodes means very few places for a
small vendor to connect to the Bearshare network.

Yvon