Loading ...
Sorry, an error occurred while loading the content.

Comments on page 963

Expand Messages
  • Eliezer S. Yudkowsky
    Let it first be said that I applaud Russell and Norvig for including section 26.3 in the book, regardless of my comments thereupon. * p. 963: The
    Message 1 of 1 , May 26, 2007
    • 0 Attachment
      Let it first be said that I applaud Russell and Norvig for including
      section 26.3 in the book, regardless of my comments thereupon.


      p. 963: "The "intelligence explosion" has also been called the
      technological singularity by mathematics professor and science fiction
      author Vernor Vinge, who writes (1993), "Within thirty years, we will
      have the technological means to create superhuman intelligence.
      Shortly after, the human era will be ended."

      Vernor Vinge coined the term "Singularity" after trying and failing to
      write a science fiction story in which the hero was smarter than
      human. Vinge found that he could not write the story, because he was
      unable to realistically envision such a hero. As Vinge later wrote
      (_True Names and Other Dangers_ short story collection, p. 47):

      "Here I had tried a straightforward extrapolation of technology, and
      found myself precipitated over an abyss. It's a problem we face every
      time we consider the creation of intelligences greater than our own.
      When this happens, human history will have reached a kind of
      singularity - a place where extrapolation breaks down and new models
      must be applied - and the world will pass beyond our understanding."

      Vinge originally coined "Singularity", not by analogy to a
      "singularity" of a function, but by analogy to the "singularity" at
      the center of a black hole, where (circa 1983) physicists' models
      broke down and gave nonsensical answers. The original "Singularity"
      was purely epistemological - it was a breakdown in your *model of*
      reality, not reality itself. It's pretty hard to extrapolate your
      model of the future (said Vinge) past the point where that model
      starts predicting the existence of minds substantially smarter than human.

      As far as I know this is Vinge's original observation, and it is
      logically distinct from I. J. Good's earlier thesis of the
      intelligence explosion. You'd have to separately argue whether or not
      an AI could recursively self-improve; versus whether or not a
      smarter-than-human AI would produce a substantial discontinuity with
      today's world (I would argue that it depends on the initial conditions
      of the AI). You can imagine A without B, B without A, neither, or both.

      Especially with regard to the "Singularity", which seems rapidly to be
      turning into a "suitcase word" a la Minsky, it is critical to keep
      distinct theses distinct. Otherwise someone may define the thesis one
      way, argue for another thesis, and present a third thesis as having
      been proven. Today Vinge's original thesis is sometimes referred to
      as the "Event Horizon" to distinguish it from other, later usages of
      the word "Singularity".


      p. 963: "Good and Vinge (and many others) correctly note that the
      curve of technological progress is growing exponentially at present
      (consider Moore's Law). However, it is quite a step to extrapolate
      that the curve will continue on to a singularity of near-infinite growth."

      Moore's Law is logically distinct from both I. J. Good's intelligence
      explosion and from Vinge's event horizon. If anything, Vinge's thesis
      argues against the indefinite continuation of Moore's Law, because it
      would be surprising if industry models developed for human
      corporations worked for predicting events after the advent of
      smarter-than-human AIs.

      Vinge's thesis is about an absolute threshold which could be
      approached by exponential progress, linear progress, logarithmic
      progress, etc. As far as I know, Moore's Law did not become
      associated with Vinge's Singularity until Vinge started trying to
      predict the time of his Singularity - a whole different ballpark from
      the core thesis itself! You need far more knowledge, and not just
      about aerodynamics, to say "Someone will build a flying machine in
      1905" instead of "Someone will build a flying machine eventually." A
      date might not be predictable even in principle. If you took the
      world from 1880 and reran the planet, the first flying machine might
      be built ten years earlier or later. It may not even be useful to
      think of an absolute level of processing power as "necessary for AI".
      Rather, the more processing power the researchers have, the less
      clever they can be and still build AI. Even this is not always true;
      if the one does not understand regularization, then building a neural
      network with a billion times as many units just implies a billion
      times as much overfitting.

      I. J. Good's thesis is that, if you can make yourself a little
      smarter, you are then able to see more ways to make yourself even
      smarter, and so a chain reaction of self-improvement occurs. This
      chain reaction does not need to continue forever, nor follow an exact
      exponential curve, in order to be of great practical significance.
      Biological neurons fire at less than 200Hz; biological axons transmit
      messages at 150 meters/second which is less than a millionth the speed
      of light; and each spike dissipates around a million times as much
      energy as the thermodynamic minimum at 300 Kelvin. The laws of
      physics definitely permit the construction of a computer at least a
      million times as fast as the human brain without shrinking the brain
      or cooling the brain. Even if the intelligence explosion tops off at
      exactly this point, the existence of minds a million times faster than
      human (never mind "a million times as smart") would <understatement
      size='huge'>be of great practical significance</understatement>.


      p. 963: "Ray Kurzweil, in The Age of Spiritual Machines (2000),
      predicts that by the year 2099 there will be "a strong trend toward a
      merger of human thinking with the world of machine intelligence that
      the human species initially created. There is no longer any clear
      distinction between humans and computers." There is even a new word -
      transhumanism - for the active social movement that looks forward to
      this future. Suffice it to say that such issues present a challenge
      for most moral theorists, who take the preservation of human life and
      the human species to be a good thing."

      The FAQ of the World Transhumanist Association defines transhumanism
      as "The intellectual and cultural movement that affirms the
      possibility and desirability of fundamentally improving the human
      condition through applied reason, especially by developing and making
      widely available technologies to eliminate aging and to greatly
      enhance human intellectual, physical, and psychological capacities."
      Cyborgs are not explicitly mentioned, either for inclusion or
      exclusion - it's about generalized persons in general.

      It is widely agreed that if a young child falls on the train tracks,
      there is a moral duty to pull them away. It is widely agreed that if
      someone of age 50 suffers from a debilitating disease that decreases
      their quality of life, it is good to cure them. Now if you have a
      logical turn of mind, you are bound to ask whether this is a special
      case of a general ethical principle which says "Life is good, death is
      bad; health is good, sickness is bad" and, if so, whether it would be
      a good thing to extend lifespan and healthspan out to 150 years, not
      just 75 years. Many people feel an instinctive shock at this, because
      it is not an accustomed idea, and they will rationalize reasons why
      150 years of health is a dangerous and subversive notion, much worse
      than 55 years of health followed by 20 years of sickness followed by
      death. But one who reads scientific history and has a sense of
      temporal perspective might remember that anesthetics and the smallpox
      vaccine were viewed with great suspicion by the
      bioethicist-equivalents of that day.

      "Transhumanist" ethics are actually simpler - can be specified with
      fewer bits - because they are consistent in their judgments; life is
      good, death is bad, health is good, sickness is bad, and there is no
      special exception to this rule when you extend lifespan and healthspan
      beyond 75, or when you use startling new technologies to get them.
      Once you see a happy, healthy person, you're done, whether they
      previously lived 40 years or 140 years, and whether they're made of
      carbon or silicon.

      So a transhumanist analytic philosopher (e.g. Nick Bostrom) would not
      say that there is anything inherently desirable about cyborging
      yourself, but would also say that there is nothing inherently
      undesirable about it. The prospect is interesting insofar as it may
      be a useful means to such normative ends as health, vigor, or
      intelligence. And while _Wired_ editors may get a kick (and extra
      sales) out of using shocking futuristic phrases like "becoming
      indistinguishable from our machines", this doesn't become desirable
      (to a transhumanist) just because it sounds shocking and futuristic;
      you'd have to make a case under utilitarian ethics. A transhumanist
      would be open to that case - they wouldn't run away, screaming,
      "Machines! Ew!" - but they would still demand that you make the
      argument under consequentialism.


      p. 963: "For the most part, it seems that robots are the protagonists
      of so many conquer-the-world stories because they represent the
      unknown, just like the witches and ghosts of tales from earlier eras.
      Do they pose a more credible threat than witches and ghosts? If
      robots are properly designed as agents that adopt their owner's goals,
      then they probably do not: robots that derive from incremental
      advances over current designs will serve, not conquer. Humans use
      their intelligence in aggressive ways because humans have some
      innately aggressive tendencies, due to natural selection. But the
      machines we build need not be innately aggressive, unless we decide to
      build them that way."

      I agree with the general thrust but its achievement is being taken too
      much for granted. If most technologies' negative repercussions tend
      to be substantially outweighed by their positive aspects, that is a
      historical generalization which implicitly takes into account the
      professional paranoia that scientists and engineers exert to keep
      things that way. A modern nuclear power plant is not safe because the
      engineers involved waved off all objections by saying that Technology
      Is the March of Human Progress, but because the engineers spent all
      night worrying about how the reactor design might fail. In the same
      way that Moore's Law would grind to a halt if all the chip-design
      researchers decided to take a vacation, technological consequences
      stop being positively skewed when engineers stop being pessimistic.

      Indeed, an AI need not share human aggressive tendencies, unless we
      design them that way; and an AI need not share human positive
      tendencies, unless we understand how to design them that way and
      successfully do so. All human beings have roughly the same cognitive
      architecture; we all have a prefrontal cortex, cerebellum, amygdala,
      etc. In the space of all possible mind designs, all human beings are
      packed into one small dot. What the term "Artificial Intelligence"
      really refers to is all the rest of mind design space; there are
      enormously more possible AIs than possible humans. To reach into that
      gigantic space, and pluck out a highly intelligent agent with a
      knowably positive impact on the world relative to our utility
      function, is a technical challenge of very high order.

      Humanity has solved high-order technical challenges before; it wasn't
      exactly easy to walk on the Moon. Here, the big problem is that it
      might be significantly easier to construct a highly intelligent agent
      that is *not* carefully shaped to a positive outcome - easier to build
      an unFriendly AI than a Friendly AI. It's hard to see how it could be
      otherwise, since the design space of highly intelligent AIs is a
      superset of the space of highly intelligent Friendly AIs. If I. J.
      Good is right about the intelligence explosion effect, this could be a
      really severe dilemma for humanity - we'd have to solve the harder
      technical challenge *first*.

      Many current AI techniques, such as gradient descent in neural
      networks, produce systems that are intelligent but to a large degree
      opaque. Evolutionary programming produces code that may partially
      match an optimization criterion, but does not match it exactly, and
      may produce wildly different behaviors in contexts outside the
      training problems. These techniques, which are still growing in
      power, are not at all well-suited to shaping a knowably Friendly AI.
      "So use Bayesian decision theory," you say, and I agree in principle.
      But there are major design challenges in building a reflective
      Bayesian agent that self-modifies and self-improves, but maintains an
      invariant optimization target relative to the outside world (a utility
      function or generalization thereof). We do not currently know how to
      do this. It will take new mathematics. Current decision theory would
      go into an infinite loop and crash if you tried to use it to calculate
      the expected utility of modifying the part of the AI that does the
      self-modifications. It's a math problem, and I think it's a solvable
      math problem, but someone has to actually solve it or there may not be
      a positive outcome for humanity.

      All this, of course, is a rather long story; see my book chapter
      "Artificial Intelligence as a positive and negative factor in global
      risk", draft online at http://singinst.org/AIRisk.pdf. But in
      summary, there are specific design challenges in ensuring a positive
      outcome - good will is not enough, there must be knowledge - and
      success on these challenges should not be taken for granted by the
      next generation of AI researchers.

      Eliezer S. Yudkowsky http://singinst.org/
      Research Fellow, Singularity Institute for Artificial Intelligence
    Your message has been successfully submitted and would be delivered to recipients shortly.