Hi everyone,

For some time it has been clear that the scripts I use for the
ALDS site ( http://struggle.net/alds ) need major improvements.
The following is a first attempt to define the features that will
be needed for a web-based message system that meets the needs of
serious public discussion. Activists who post need greater
control over the format of their post. In addition, the process
of various posts competing for attention needs to be more
organized and more democratic (and less reliant on either
chronology or a single web page editor deciding which
installments or essays are deserving of enhanced attention).

The design philosophy here is influenced by two phenomena that
are currently kicking butt on the web: weblogs and wikis.

Documentation for this project will eventually include:
(a) a set of "use cases"
(b) a functional spec
(c) a technical spec
(d) a set of test cases.

Everything here is subject to revision and your suggestions are
needed. The temporary name for this project will be "red node".
The term "node" refers to the fact that each post will also be a
separate mini-forum with its own page on the same basis as its
parent post (ie: the post to which it is replying) or its
children (ie: the posts which reply to it). Eventually (maybe
with help from you) we can think of a better name to symbolize
the militant use of this bulletin board system (ie: something
like "fist").

Major funtional areas:
-------------
I. Simple Stupid Markup Language
II. Filtering
III. Rating
IV. Replying to posts
V. Future features

------------------------------------------------------------
I. Simple Stupid Markup Language
------------------------------------------------------------

Activists who post will be able to make use of simple tags or
similar markers so that their posts will have more features than
plain text.

1. The html form will have text boxes for: (a) title,
(b) subtitle, (c) body, (d) author, and (e) author home page,
weblog and/or email address, (f) teaser (a teaser is a line
in small font that floats above the title).

2. The body of the post will have the following features:
(g) subheads, (h) bold text, (i) italic text, (j) quoted text
sections (similar to html style blockquotes but probably with a
colored (light green or blue or yellow) background), (k) pull
quotes (similar to quoted text sections but with enlarged
boldface text instead of a colored background), (l) sidebar
text (similar to quoted text but with gray background), (m)
footnotes (ie: that are linked to from the main body, and which
link back, when rendered in html format), (n) html-style ordered
and unordered lists, (o) paragraph separation, (p) links within
the text body.

3. Eventually provision will be made for certain kinds of tables
and images--but not initially. In version 1.0, these will be
added manually to the html formatted version that is
automatically generated (if tables or images are really
necessary).

4. Anyone who makes a post will be able to review their post
before it becomes permanent. This is particularly important
since the formatting abilities make it likely that many mistakes
will be made.

5. It is unclear at this time how best to handle situations where
the author wants to make changes to a post after it has become
permanent. There are arguments on both sides of this question:
(a) in some cases it may be necessary and (b) it could create
confusion if the post is modified after readers read and comment
on it. One possible compromise would be to allow changes to be
made if no one has yet replied to the post. Another possible
compromise would be to allow changes but to keep a permanent
viewable record of all versions together with their timestamps.

-----------------------------------------------------
Technical notes for "simple stupid markup language":
-----------------------------------------------------

(Technical notes about features in the functional spec will
eventually be branched off into a separate document called the
technical spec--but for now--are included, as here, as separate
sections in the functional spec.)

1. An idea of what kinds of features will be needed can be seen
by looking at installment 5 or essay 160 of the ALDS here:
http://Struggle.net/alds/part_05_content.htm
http://Struggle.net/alds/essay_160_content.htm

2. The post will be stored using xml-like tags: <title>,
<subtitle>, etc. The rendering engine will transform the xml
version of the post into two versions: (a) html and (b) plain
text. The plain text version will be for use with email.
Eventually, this system will automatically send (and receive)
email to (and from) readers (and posters). Also, eventually a
third type of format can be generated which is optimized for
printing. This format will be like the html format but will use a
serif font (ie: Times New Roman) rather than a sans-serif font
(ie: Arial).

3. In the plaintext format boldface words or phrases will begin
and end with the ~enya~ character while italized words or phrases
might begin and end with the _underscore_ character. Boldface
can be indicated, in simple stupid markup language, with the
familiar <b> and </b> tags or like [[this]] while italic can use
<i> and </i> or {{this}}.

4. Other allowed html tags will include: <ul>, <ol>, <li>, <br>

5. Pseudo-html tags will include: <quote>, <pullquote>,
<footnote_link/x> and <footnote_text/x> (where "x" is the
footnote number)

6. Paragraphs will be indicated with the use of a blank line
separating paragraphs.

7. All paragraphs (and subheads) will be marked with a paragraph
number that is automatically generated and which will be
displayed (in the html version) in such a way that it does not
interfere with the presentation of the text. An example of how
this might look can be seen here:
http://Leninism.org/some/76x.htm . Paragraph numbers will
greatly assist the ability of posters to reference particular
paragraphs of a text on which they chose to comment or link
(each paragraph will be an html anchor so that it will be
possible to link directly to individual paragraphs). It is
unclear at this time if it will be worthwhile to display
paragraph numbers for the email version.

8. Links will be handled two ways: (a) web URL's (ie: anything
that begins with http:// ) will be automatically converted into
links (except that trailing periods followed by a space--which
probably indicate the end of sentence--will not be made part of
the link) and (b) we may want to allow posters to substitute text
for the link similar to the way the <a href> tag works in html.
However we will probably not allow direct use of the <a href> tag
because it is very easy for posters to make syntax errors with it
(for example leaving out quote marks, etc) and it would require a
lot of code (and testing) to check for this. So if we allow for
text links we will probably require posters to use something like
the following:

<link>
<url>struggle.net/alds</url>
<text>anarcho-leninist debate on the state</text>
</link>

Our scripts will check for closing tags for <link>, <url>, <text>
to reduce the error rate and all links will be automatically
rendered (using iframes) on the review page in order to reduce
the errors that are inevitable when posters enter web addresses
based on faulty human memory.

It should also be simple and easy to link to any other post in
this system. I am thinking of syntax similar to this:

<post>102-2034</post>

for linking to a post (ie: in this case post # 2034
in forum # 102) and similar to this:

<post>102-2034-46</post>

for linking to a specific paragraph
(ie: in this case paragraph # 46) within a post.

Links which go to sites outside of the Red Node message system
will result in a new browser window popping up. (Note: some
people like a separate browser window appearing in this kind of
situation and some do not. This is _not_ the same as pop-up
ads--which are extremely obnoxious. The advantage of a separate
browser window is that you can easily follow the link without
losing the browser window containing the original post which you
may not have finished reading.)

------------------------------------------------------------
II. Filtering
------------------------------------------------------------

The ability of anonymous activists to post over the web creates
problems for any forum that aspires to develop a high
signal-to-noise ratio. Problems include:
(a) neo-nazi and organized racist groups which actively
seek out and attempt to establish a disruptive presence
on progressive bulletin boards,
(b) people who make threats or similar conduct that has
potential to create legal problems (for example the owner
of the popular militant anarchist site at
http://RaiseTheFist.com
was sent to jail on trumped up charges relating, in part, to
the fact that someone made a post to his site which linked
to a site with instructions on how to make a bomb),
(c) organized efforts at disruption from right-wing sites like
http://FreeRepublic.com which have been successful in
clogging up Seattle Indymedia with right-wing flamebait
(d) the aggressively clueless and excessively stupid
who spam, troll or are emotionally immature.

The solution to these problems is:
(a) limited posting frequency (based on capture of IP checksum)
(b) filtering and
(c) rating.

IP checksum capture and filtering is described in this section.

1. Limits based on IP address and time
----------------------------------------

The first line of defense is an IP address capture system that
limits posters to one post per hour per forum. This simple
measure, by itself, reduces the most common types of abuse by
approximately 90 percent or more and has been successfully used
for more than a year at the web-based forum at
http://communism.com (a site that invites a lot of abuse from
bored high school kids).

security/privacy note
----------------------

Security/privacy issues must be considered if we capture and
store IP addresses. An IP address (together with info from your
ISP) will usually be enough to identify who you are. The FBI
attempted to seize IP addresses from Indymedia following
political protests in Toronto (I think it was in 2000) and this
is the reason that Indymedia no longer stores IP addresses.

But we need to store some kind of data to reduce system abuse or
our signal-to-noise ratio will fall to the level found on
Indymedia (ie: too shitty for a serious forum). Our solution is
that our scripts will not store IP address directly (we do not
want to make it too easy for the political police to look at our
records and see who is posting). Hence we will use the
equivalent of a checksum, in which the 4 billion possible IP
addresses are mapped to approximately ten thousand possible
checksums and each OS/browser combination is similarly mapped to
a reduced data space. The political police (if they employ
experts to study our scripts) would still be able to cross-check
a suspected IP address against our checksums--but if they did not
have a specific IP address to cross-check -- they would have a
difficult time working backwards (ie: going on a fishing
expedition) because each IP checksum that we store will map to
approximately four hundred thousand IP addresses.

Of course if the political police take a strong enough interest
in the site (something that may happen within the next ten years)
they could simply tap the site's internet line and collect the
full IP address for each poster and each reader. But this is
outside of our control and is unrelated to our storage of IP
checksums. There is nothing that we can do about heavy-duty
surveillance--except build the site by having so many readers and
posters that the political police would find it difficult, when
the time comes, to harass or intimidate posters.

Note: eventually we will have a registration system. Activists
who demonstrate that they are mature will be able to post more
often than once per hour.

2. Filtering
-----------

Posts for each forum will be posted to the alpha staging page
(there will be an alpha staging page for each of the many forums
as well as a ~global~ alpha staging page which is equivalent to
the all the local alpha staging pages combined). The alpha
staging page for each forum will probably have a public address
(so readers will be able to see it live if they want).

Our filter volunteers (ie: for now, me, eventually others may
wish to help out) will check the global alpha staging page
approximately every 24 hours. All posts will be scanned for (a)
evidence of promotion of neo-nazi or racist groups and (b)
threats of any kind. Posts representing this kind of abuse will
be transferred to our alpha_reject_A (ie: organized racists) and
alpha_reject_B (ie: threats) pages (these pages will not be
public--we will keep these alpha rejects on password protected
pages for study and training). Hence neo-nazi posts and threats
will not be public for more than 24 hours (on average) and will
not make it past our alpha staging pages.

24 hour response time for alpha filtering is good enough. It is
a test of our militancy to remove, within 24 hours, all ravings
by organized racists from our public alpha staging pages. 24
hours is also satisfactory as far as removing threatening posts:
the political police would have great difficulty persuading a
jury that we were responsible for the content of any post that we
removed within 24 hours.

Posts which are free of these two categories of abuse will be
transferred to the beta staging pages (local and global). The
global beta page will be periodically checked by our volunteers
(ie: me, for now) and posts which are either insincere,
excessively stupid, etc will be transferred to the beta_reject
pages (which will be permanently public and advertised as the
pages for people who missed the cluetrain--sometimes it can be
amusing and educational for readers to look at all the collected
rejects).

Note that alpha rejection is different than beta rejection in two
main ways: (a) alpha rejection can be done by volunteers with
only a small amount of training and (b) alpha rejection needs to
be done fairly quickly (ie: 24 hours on average). Beta rejection
decisions require a greater amount of judgement than alpha
rejections since beta rejections tend to be more of a gray area
and beta rejection judgements will often be somewhat arbitrary.

One example of this distinction is that alpha rejection will
filter out neo-nazis while beta rejection will handle the
right-wing jerks from http://FreeRepublic.com as well as the
emotionally immature (including making decisions, on a case by
cases basis, whether to simply filter the post--or to reply to it
in an effort to engage the person who posted it).

Another example of the distinction between alpha and beta
filtering concerns handling of posts which may express borderline
racist views. Such issues are sometimes a gray area--and would
be considered a beta judgement call. On the other hand posts
which pledge allegiance to hitler or promote the national front,
white power or racial purity are fairly easy to identify. These
are alpha calls.

For these reasons it makes sense to separate the alpha and beta
rejection functions and manage them separately. As this system
develops each forum "owner" will decide what policy to follow for
beta rejection (ie: to let beta filtering be done by our global
volunteers--or to handle it locally--or not to do beta filtering
at all). Alpha filtering, on the other hand, is mandatory for
all forums using our red node scripts. No (a) neo-nazi ravings
or (b) threats which are legally actionable will make it past our
alpha filters.

Posts which pass the beta filter will be transferred to the gamma
pages. This is where readers will rate the posts. It is also
possible that, on some forums, the gamma pages will be the same
page as the beta pages. This might be useful if the beta
filtering (which requires a closer and more time-consuming
reading of each post) takes too long. We will get a more
realistic picture of what is practical as we gain experience.

Each forum will have its own gamma pages. It is not practical
for all posts to be on a single page so the posts will be
arranged in chronological pages, with between ten to fifty posts
per page (most recent post on top in typical blog fashion--and
the most recent page being the first page you see--and which
links to the next older page, etc). Not all of a lengthy post
will be readable on the gamma page. For example it may be that
only the first 600 words would be on the gamma page--and the rest
of the post would be visible by following a link to a page that
is devoted to that particular post and which displays the entire
post (and also shows the responses to that post--see below).

It will also be possible to aggregate gamma pages for different
forums. The need for this was demonstrated by the ALDS site. On
that site there is a different forum for (a) comments, (b)
questions, (c) links and (d) for _each_ of the 16 installments or
essays. It can be advantageous to have a separate mini-forum for
each essay because this allows readers to view in one place, for
example, all the responses to essay # 153. However it is also
advantageous to have a single place a reader can go to check
whether there is anything new in _any_ of the nearly twenty ALDS
forums--in particular when the total number of postings for all
forums combined averages less than one per week.

------------------------------------------------------------
III. Rating
------------------------------------------------------------

Ratings are done by our readership (making use of the same IP
checksum capturing used to limit posting to one post per forum
per hour). It will still be possible for determined people to
vote more than once for a particular rating for a post they
intensely like or dislike--but the IP checksum capturing will
reduce abuse by more than 90 percent (and the remaining abuse is
usually not too difficult to identify) and as the system gains a
larger readership the abuse will be outweighed by the ratings
made by readers who aren't trying to cheat the system.

For each post readers will be able to vote either: excellent,
good, ok, poor or complete bullshit (ie: 4, 3, 2,1 and 0). The
average rating for each post will be displayed in an iframe
window (the use of the iframe window will eliminate the need to
regenerate the entire page each time the average rating changes
for any of the posts on that gamma page). Readers will be able
to select the rating cutoff level somewhat similar to how it is
done on slashdot. There will be different sets of gamma pages
corresponding to different rating cutoff levels. For example, a
reader may choose to see only posts rated 2 or higher (or 2.5 or
higher, or 3.0 and higher, etc). Currently, I am planning for
six distinct sets of gamma pages (for cutoff levels at: 0, 1, 2,
2.5, 3 and 3.5). The generation of the six sets of gamma pages
would probably take place about every hour or so.

The gamma page sets would only be generated once every hour
(rather than continuously, in real time, whenever the average
rating on a post fluctuates above or below a certain level)
because this will reduce the load on our system and may eliminate
the need for us to hook up to a database. It is easier to code
and test scripts which don't use a database and which create
static pages. If this system is successful--then eventually
competent coders with time on their hands will create a better,
more robust version of it that connects to a live database and
has other neat features and improvements (ie: like collaborative
filtering). But we want something that is as simple as
possible--and static pages are simpler.

Furthermore an hour timeframe is probably fine for us because
the timeframe for calm, serious, deliberative discussion on
weighty issues tends to be longer than the fast-paced discussion
on a site like slashdot (which is maintained by hard-core geeks
who know what they are doing).

Once we have a registration system we may (again--like slashdot)
add a point to the rating of a post if it is made by someone who
is registered. This will be an incentive to people to register.
We may also sort out some way to add to the rating of a post if
the post attracts a lot of replies (see below) although I have
not yet given much thought to what sort of rules would be useful
in relation to this. One potential way to handle this would be
to create a separate page for each month listing the most popular
posts as determined by the number of responses that they
generated.

------------------------------------------------------------
IV. Replying to posts
------------------------------------------------------------

This is where matters become interesting because each post will
become, in effect, its own forum administered (once we have a
registration system) by the author of the post. This is called
recursion and it may allow our posting system to have a high
degree of flexibility.

Each post will have a button that readers can click if they want
to reply. Each post will have a small iframe window showing how
many replies there have been and the titles of each reply (and
the rating that the author of each reply gave to the post--using
happy/frowning faces or thumbs up/down icons). If the reader
clicks to bring up the page which is devoted to that post--the
reader will view the gamma page for replies to that post.

Also--it will be an option that the replies to a post will also
appear on the same set of gamma pages where the post appears.
So, for example, if post B replies to post A, then post B will
not only appear on the page devoted to post A--but will also
appear above post A on the gamma page where the first 600 words
of post A are displayed. This would insure that replies to posts
would not be lost but would be visible on the same basis as other
posts higher up the tree.

------------------------------------------------------------
V. Future features
------------------------------------------------------------

If the Red Node project is successful (ie: is used by dozens or
hundreds of people and is generally considered useful) other
features can be added as time allows:

1. A registration system to give preferential treatment (allow to

post more often, have a rating advantage, control their own
forums) to registered users.

2. collaborative filtering--in which you can make use of the
ratings made by people who tend to vote on ratings the same way
that you do (this is only possible with registered users)

3. Replies at the paragraph level--so that you can look in the
"margins" of the paragraphs of a length post you are reading--and
see if there are any replies to that particular paragraph

4. The forum may eventually be usable via email. The first stage
would be to give readers the option of having replies to their
posts (or any replies to a forum in which they are interested)
sent to them via email. The second stage (requiring more
technology) would allow posters to submit posts via email.

5. Simple stupid markup language could be developed to allow for
the use of graphics (similar to the link tags) like this:

<image>
<url>struggle.net/alds</url>
<text>anarcho-leninist debate on the state</text>
</image>

In this case the <text> tag will be used to indicate the caption.

6. Simple kinds of tables (similar to way that some wiki's allow)
could be demarcated like this:

|| this represents || a table || that is three cells ||
|| wide || and two cells || tall ||

Tables will be useful when we want to list information is rows
and columns. It is unclear at this time which table format would
be most useful. One example of a possible table format can be
seen here: http://struggle.net/anti-war/table-test2.htm

7. Also (like some wikis) two levels of subhead could be allowed.
They would appear in the simple stupid markup "source"
(demarcated with either two or three dashes at the beginning and
end of the subhead) like this:

-- this would be a subhead --

or like this:

--- a subhead that uses a somewhat smaller font ---

Using the principles that have been pioneered in wiki's -- we
would like the source to as readable as possible and resemble
the final html rendition.

8. Security: If this project becomes successful it could find
application in many kinds of situations and, as a result of wide
usage and recognition, would eventually attract interest from
organized racists or right-wing vigilantes (directly or
indirectly assisted by the budgets and technical expertise of the
political police). As such, the project and associated web site
would become subject to hacking attacks of varying degrees of
sophistication. We keep in mind that, at this stage, nearly all
websites are vulnerable to attack from skilled (or often, even
unskilled, so-called "hackers"). Our main line of defense will
be to back up the site regularly and mirror the site content on a
set (if necessary a large set) of other sites. This is another
advantage of creating static pages: the site content can be
easily mirrored to sites that use any operating system because
server side scripts are only needed to make a post or to rate a
post--but not to view postings.

9. If the principles developed in this site prove valuable these
principles can then be used by activists with deeper and more
solid techical skills and implemented in a more professional and
robust way. Such a development would represent the ultimate
success of this project.

<>