Let it first be said that I applaud Russell and Norvig for including
section 26.3 in the book, regardless of my comments thereupon.
p. 963: "The "intelligence explosion" has also been called the
technological singularity by mathematics professor and science fiction
author Vernor Vinge, who writes (1993), "Within thirty years, we will
have the technological means to create superhuman intelligence.
Shortly after, the human era will be ended."
Vernor Vinge coined the term "Singularity" after trying and failing to
write a science fiction story in which the hero was smarter than
human. Vinge found that he could not write the story, because he was
unable to realistically envision such a hero. As Vinge later wrote
(_True Names and Other Dangers_ short story collection, p. 47):
"Here I had tried a straightforward extrapolation of technology, and
found myself precipitated over an abyss. It's a problem we face every
time we consider the creation of intelligences greater than our own.
When this happens, human history will have reached a kind of
singularity - a place where extrapolation breaks down and new models
must be applied - and the world will pass beyond our understanding."
Vinge originally coined "Singularity", not by analogy to a
"singularity" of a function, but by analogy to the "singularity" at
the center of a black hole, where (circa 1983) physicists' models
broke down and gave nonsensical answers. The original "Singularity"
was purely epistemological - it was a breakdown in your *model of*
reality, not reality itself. It's pretty hard to extrapolate your
model of the future (said Vinge) past the point where that model
starts predicting the existence of minds substantially smarter than human.
As far as I know this is Vinge's original observation, and it is
logically distinct from I. J. Good's earlier thesis of the
intelligence explosion. You'd have to separately argue whether or not
an AI could recursively self-improve; versus whether or not a
smarter-than-human AI would produce a substantial discontinuity with
today's world (I would argue that it depends on the initial conditions
of the AI). You can imagine A without B, B without A, neither, or both.
Especially with regard to the "Singularity", which seems rapidly to be
turning into a "suitcase word" a la Minsky, it is critical to keep
distinct theses distinct. Otherwise someone may define the thesis one
way, argue for another thesis, and present a third thesis as having
been proven. Today Vinge's original thesis is sometimes referred to
as the "Event Horizon" to distinguish it from other, later usages of
the word "Singularity".
p. 963: "Good and Vinge (and many others) correctly note that the
curve of technological progress is growing exponentially at present
(consider Moore's Law). However, it is quite a step to extrapolate
that the curve will continue on to a singularity of near-infinite growth."
Moore's Law is logically distinct from both I. J. Good's intelligence
explosion and from Vinge's event horizon. If anything, Vinge's thesis
argues against the indefinite continuation of Moore's Law, because it
would be surprising if industry models developed for human
corporations worked for predicting events after the advent of
Vinge's thesis is about an absolute threshold which could be
approached by exponential progress, linear progress, logarithmic
progress, etc. As far as I know, Moore's Law did not become
associated with Vinge's Singularity until Vinge started trying to
predict the time of his Singularity - a whole different ballpark from
the core thesis itself! You need far more knowledge, and not just
about aerodynamics, to say "Someone will build a flying machine in
1905" instead of "Someone will build a flying machine eventually." A
date might not be predictable even in principle. If you took the
world from 1880 and reran the planet, the first flying machine might
be built ten years earlier or later. It may not even be useful to
think of an absolute level of processing power as "necessary for AI".
Rather, the more processing power the researchers have, the less
clever they can be and still build AI. Even this is not always true;
if the one does not understand regularization, then building a neural
network with a billion times as many units just implies a billion
times as much overfitting.
I. J. Good's thesis is that, if you can make yourself a little
smarter, you are then able to see more ways to make yourself even
smarter, and so a chain reaction of self-improvement occurs. This
chain reaction does not need to continue forever, nor follow an exact
exponential curve, in order to be of great practical significance.
Biological neurons fire at less than 200Hz; biological axons transmit
messages at 150 meters/second which is less than a millionth the speed
of light; and each spike dissipates around a million times as much
energy as the thermodynamic minimum at 300 Kelvin. The laws of
physics definitely permit the construction of a computer at least a
million times as fast as the human brain without shrinking the brain
or cooling the brain. Even if the intelligence explosion tops off at
exactly this point, the existence of minds a million times faster than
human (never mind "a million times as smart") would <understatement
size='huge'>be of great practical significance</understatement>.
p. 963: "Ray Kurzweil, in The Age of Spiritual Machines (2000),
predicts that by the year 2099 there will be "a strong trend toward a
merger of human thinking with the world of machine intelligence that
the human species initially created. There is no longer any clear
distinction between humans and computers." There is even a new word -
transhumanism - for the active social movement that looks forward to
this future. Suffice it to say that such issues present a challenge
for most moral theorists, who take the preservation of human life and
the human species to be a good thing."
The FAQ of the World Transhumanist Association defines transhumanism
as "The intellectual and cultural movement that affirms the
possibility and desirability of fundamentally improving the human
condition through applied reason, especially by developing and making
widely available technologies to eliminate aging and to greatly
enhance human intellectual, physical, and psychological capacities."
Cyborgs are not explicitly mentioned, either for inclusion or
exclusion - it's about generalized persons in general.
It is widely agreed that if a young child falls on the train tracks,
there is a moral duty to pull them away. It is widely agreed that if
someone of age 50 suffers from a debilitating disease that decreases
their quality of life, it is good to cure them. Now if you have a
logical turn of mind, you are bound to ask whether this is a special
case of a general ethical principle which says "Life is good, death is
bad; health is good, sickness is bad" and, if so, whether it would be
a good thing to extend lifespan and healthspan out to 150 years, not
just 75 years. Many people feel an instinctive shock at this, because
it is not an accustomed idea, and they will rationalize reasons why
150 years of health is a dangerous and subversive notion, much worse
than 55 years of health followed by 20 years of sickness followed by
death. But one who reads scientific history and has a sense of
temporal perspective might remember that anesthetics and the smallpox
vaccine were viewed with great suspicion by the
bioethicist-equivalents of that day.
"Transhumanist" ethics are actually simpler - can be specified with
fewer bits - because they are consistent in their judgments; life is
good, death is bad, health is good, sickness is bad, and there is no
special exception to this rule when you extend lifespan and healthspan
beyond 75, or when you use startling new technologies to get them.
Once you see a happy, healthy person, you're done, whether they
previously lived 40 years or 140 years, and whether they're made of
carbon or silicon.
So a transhumanist analytic philosopher (e.g. Nick Bostrom) would not
say that there is anything inherently desirable about cyborging
yourself, but would also say that there is nothing inherently
undesirable about it. The prospect is interesting insofar as it may
be a useful means to such normative ends as health, vigor, or
intelligence. And while _Wired_ editors may get a kick (and extra
sales) out of using shocking futuristic phrases like "becoming
indistinguishable from our machines", this doesn't become desirable
(to a transhumanist) just because it sounds shocking and futuristic;
you'd have to make a case under utilitarian ethics. A transhumanist
would be open to that case - they wouldn't run away, screaming,
"Machines! Ew!" - but they would still demand that you make the
argument under consequentialism.
p. 963: "For the most part, it seems that robots are the protagonists
of so many conquer-the-world stories because they represent the
unknown, just like the witches and ghosts of tales from earlier eras.
Do they pose a more credible threat than witches and ghosts? If
robots are properly designed as agents that adopt their owner's goals,
then they probably do not: robots that derive from incremental
advances over current designs will serve, not conquer. Humans use
their intelligence in aggressive ways because humans have some
innately aggressive tendencies, due to natural selection. But the
machines we build need not be innately aggressive, unless we decide to
build them that way."
I agree with the general thrust but its achievement is being taken too
much for granted. If most technologies' negative repercussions tend
to be substantially outweighed by their positive aspects, that is a
historical generalization which implicitly takes into account the
professional paranoia that scientists and engineers exert to keep
things that way. A modern nuclear power plant is not safe because the
engineers involved waved off all objections by saying that Technology
Is the March of Human Progress, but because the engineers spent all
night worrying about how the reactor design might fail. In the same
way that Moore's Law would grind to a halt if all the chip-design
researchers decided to take a vacation, technological consequences
stop being positively skewed when engineers stop being pessimistic.
Indeed, an AI need not share human aggressive tendencies, unless we
design them that way; and an AI need not share human positive
tendencies, unless we understand how to design them that way and
successfully do so. All human beings have roughly the same cognitive
architecture; we all have a prefrontal cortex, cerebellum, amygdala,
etc. In the space of all possible mind designs, all human beings are
packed into one small dot. What the term "Artificial Intelligence"
really refers to is all the rest of mind design space; there are
enormously more possible AIs than possible humans. To reach into that
gigantic space, and pluck out a highly intelligent agent with a
knowably positive impact on the world relative to our utility
function, is a technical challenge of very high order.
Humanity has solved high-order technical challenges before; it wasn't
exactly easy to walk on the Moon. Here, the big problem is that it
might be significantly easier to construct a highly intelligent agent
that is *not* carefully shaped to a positive outcome - easier to build
an unFriendly AI than a Friendly AI. It's hard to see how it could be
otherwise, since the design space of highly intelligent AIs is a
superset of the space of highly intelligent Friendly AIs. If I. J.
Good is right about the intelligence explosion effect, this could be a
really severe dilemma for humanity - we'd have to solve the harder
technical challenge *first*.
Many current AI techniques, such as gradient descent in neural
networks, produce systems that are intelligent but to a large degree
opaque. Evolutionary programming produces code that may partially
match an optimization criterion, but does not match it exactly, and
may produce wildly different behaviors in contexts outside the
training problems. These techniques, which are still growing in
power, are not at all well-suited to shaping a knowably Friendly AI.
"So use Bayesian decision theory," you say, and I agree in principle.
But there are major design challenges in building a reflective
Bayesian agent that self-modifies and self-improves, but maintains an
invariant optimization target relative to the outside world (a utility
function or generalization thereof). We do not currently know how to
do this. It will take new mathematics. Current decision theory would
go into an infinite loop and crash if you tried to use it to calculate
the expected utility of modifying the part of the AI that does the
self-modifications. It's a math problem, and I think it's a solvable
math problem, but someone has to actually solve it or there may not be
a positive outcome for humanity.
All this, of course, is a rather long story; see my book chapter
"Artificial Intelligence as a positive and negative factor in global
risk", draft online at http://singinst.org/AIRisk.pdf.
summary, there are specific design challenges in ensuring a positive
outcome - good will is not enough, there must be knowledge - and
success on these challenges should not be taken for granted by the
next generation of AI researchers.
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence