Loading ...
Sorry, an error occurred while loading the content.

Re: New paper on why modules evolve, and how to evolve modular artificial neural networks

Expand Messages
  • stephane.doncieux
    Hi, Congratulations to JB, Jeff and Hod for their great work on the evolution of modularity ! The length and multiplicity of the comments undoubtedly show the
    Message 1 of 30 , Feb 20, 2013
    • 0 Attachment
      Hi,

      Congratulations to JB, Jeff and Hod for their great work on the
      evolution of modularity !

      The length and multiplicity of the comments undoubtedly show the value
      of this work. I would like to comment on the debate between encoding bias and fitness manipulations that this paper has provoked in the mailing list (a debate that actually goes beyond the connection cost objective for modularity).

      After having spent a long time on encodings, I find it fruitful to
      work on selection pressures. We have actually worked on the definition
      of such fitness pressures and have proposed objectives to promote
      diversity (Behavioral diversity [1]), generalization [2], behavior
      consistency [3,4] or to cross the reality gap with robots [5]. I
      don't want to say that it is the only way to solve the problems that
      these objectives are designed for, nor that it is the best way to do
      it, but I will just mention the advantages I find with such an
      approach relative to work on the encoding.

      The great advantage of the works on selection pressures is that it is
      highly localized. The contribution is limited to the definition of an
      objective that is then easy to integrate to a third party work than
      when it comes with its own specific selection algorithm and
      encoding. Such work does not highly depend on an encoding, nor on a
      particular selection algorithm (as long as it is a multi-objective
      algorithm). It is trivial to make comparisons with-without a
      particular objective and all these works are, at least from a
      technical point of view, compatible. You can then use, besides a
      goal-oriented objective, the connection cost objective for modularity together with a behavioral diversity or novelty objective to increase the exploration. What is even more interesting in my opinion, is that
      anyone interested in optimizing complex neural networks or graphs and
      that may have his own encoding for whatever reasons (to be
      biologically plausible for instance) can use these objectives in his
      own work with very few, if any, technical modifications. It represents
      then a lower investment and makes such kind of work more prone to be
      used outside of our own field, it is at least my opinion on the
      subject. Likewise, all the works mentionned above rely on NSGA-II,
      i.e. a classic multi-objective evolutioanry algorithm. Any
      new and more efficient MOEA can by used without any particular
      technical problem.

      To come back to the performance issue, in all of our experiments,
      using a simple direct encoding with two objectives
      (fitness+behavioral diversity) gave better results than NEAT, no
      matter how we define behavioral diversity [1]. What is nice with these
      results, is that it is not incompatible with NEAT. A
      multi-objective version of NEAT can exploit the benefits gained with
      behavioral diversity (it has actually been proposed in a GECCO paper
      by Moriguchi and Honiden). A quite surprising result for us is that a
      simple direct encoding can solve a complex sequential tasks (ball
      collecting task) with a discrete and simple fitness function (a number
      of collected balls) together with a simple behavioral diversity
      objective (see a video here: http://youtu.be/Trj0_A1ZfNo).

      Of course, there are limitations. If you use more than three
      objectives, for instance, the performance of multi-objective
      algorithms is known to be very low. This is a limitation that people
      from the multi-objective optimization community are aware of. They
      work on it and many-objectives optimization algorithms will probably
      be released in a near future (some have already been proposed
      actually). Likewise, as pointed out by Jeff, the dead weights
      mentionned by Ken are really a consequence of current multi-objective
      evolutionary algorithms inner workings. Several adaptations can be
      proposed, as the "probabilistic pareto dominance" developped by Jeff
      and Jean-Baptiste or any other preference or thresholding system. It
      is not an inherant limitation of these approaches, just a proof that
      there is still some work to do in multi-objective optimization
      field. It should be noted that this is not a problem of our community
      only. Multi-objective optimization is a very dynamic field of research
      so we can expect solutions to these problems in a forthcoming future.

      So to conclude, I agree that it is important to work on the encoding,
      but my point is that it is also worth to work on selection pressures. The nice thing is
      that the works on both aspects are quite complementary.


      Best,

      Stephane Doncieux

      [1] Mouret, J.-B. and Doncieux, S. (2012). Encouraging Behavioral Diversity in Evolutionary Robotics: an Empirical Study.
      Evolutionary Computation. Vol 20 No 1 Pages 91-133.

      [2] Pinville, T. and Koos, S. and Mouret, J-B. and Doncieux,
      S. (2011). How to Promote Generalisation in Evolutionary Robotics: the
      ProGAb Approach. GECCO'11. Pages 259--266.

      [3] Ollion, Charles and Doncieux, Stéphane (2012). Towards Behavioral Consistency in Neuroevolution.
      From Animals to Animats: Proceedings of the 12th International
      Conference on Adaptive Behaviour (SAB 2012), Springer, publisher.

      [4] Ollion, C. and Pinville, T. and Doncieux, S. (2012). With a little
      help from selection pressures: evolution of memory in robot
      controllers. Proc. Alife XIII.

      [5] Koos, S. and Mouret, J.-B. and Doncieux, S. (2013). The
      Transferability Approach: Crossing the Reality Gap in Evolutionary
      Robotics. IEEE Transactions on Evolutionary Computation. Vol 17 No 1 Pages 122 - 145 .



      --- In neat@yahoogroups.com, Jeff Clune <jclune@...> wrote:
      >
      > Hello Ken,
      >
      > Please see my inlined comments.
      >
      > >
      > > Hi Jeff, I wanted to follow up on the "dead weight" concept I brought up, because I feel I may not have made it entirely clear what that is. In a multiobjective formulation, if one of the objectives is low connectivity, then you can dominate on that objective by having extremely low connectivity and no other saving grace whatsoever. In other words we are talking about completely nonfunctional and essentially inert garbage that does well on that one objective by effectively being brain dead. In effect, you have created a special "niche" for brain-dead networks with radically low connectivity.
      > >
      > >
      > That's true. Of course, you could choose to have a cutoff to minimize this problem. For example, you could not give credit to any orgs that do not have a path from inputs to outputs, or that have a total connectivity below X, or some other solution (including a cutoff that changes over time).
      >
      > > Perhaps early on this niche is a fruitful point of departure for better places, but the problem is that this niche will never go away. You will always have this protective pocket for brain-dead low-connectivity networks for as long as evolution runs. In fact it's a very attractive niche because it's so easy - you don't have to worry about performance and simply need to keep your structure down. So these types of inert blobs will be around forever.
      > >
      > > You suggest that my concern about this "unfit" niche is not consistent with my support of novelty search, but novelty search is not really analogous. Novelty search is about constantly searching for *new* departure points. Your dead-weight niche is about keeping around the same bottom-level junk forever. Novelty search would quickly tire of such a black hole and abandon it (it doesn't stay novel). But multiobjective search will embrace it for eternity.
      > >
      > >
      > That's true, but on many complex/multi-dimensional tasks there are often infinitely many ways to be uninteresting, yet novel. For example, in your unenclosed map in the NS journal paper, only a few runs solve the problem (the same as random search), likely because of the permanent dead-weight of an infinite number of places to end up that are far away from the target.
      >
      > > Nature doesn't have anything analogous either, which means there is at least some evidence that the "fitness bias" analogy with nature is not lining up perfectly. You might point to the continuing existence of single-celled organisms as something similar to the perpetual dead-weight in this formulation, but they aren't really analogous because single-celled organisms are functional - they retain the ability to make copies of themselves and continue to evolve in their own right - while the low-connectivity deadweight maintains no capability whatsoever. On the other hand, suspiciously, as in nature, nothing similar to such a deadweight niche is perpetuated by a biased encoding.
      > >
      > >
      > That's also true, but that fault does not lie with the fitness cost concept, it lies with the fact that multi-objective algorithms, which are the cause of the dead weight, do not perfectly analogize to nature. They're just better than a weighted sum for other reasons, but the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue.
      >
      > > Doesn't it seem a little strange that the price we have to pay to obtain modular structure is to maintain a perpetual dead pool of genetic junk? Note that it doesn't suggest that such a system won't work in some cases, but it's inelegant enough to raise questions about the best realization of the concept..
      > >
      >
      > All I think that calls into question is the optimality of multi-objective algorithms when you don't want the extreme of one objective. But that problem almost always occurs in multi-objective algorithms, so your really indicting the whole field of MOEA instead of our approach of using a fitness penalty instead of a biased encoding, no?
      >
      > >
      > > Also on the analogy with nature, while your argument for fitness pressure in nature based on the size of the head is logically possible (and the kind of argument that is probably attractive to a lot of people), it's only speculative. In fact what little evidence there is on such a speculative issue (i.e. whether a baby's head might fit through a mother's pelvis in some alternate evolutionary path) doesn't support the idea that size is the main issue here. After all, whale brains are far bigger than human brains and I'm sure they are also dominated by local connectivity.
      > >
      > Yes, but they float in water! You couldn't have a creature with that big of a brain on land (I speculate, not being an expert in these things). Interesting side note: I believe whale brains are pretty tiny as a fraction of body size compared to primates, and (for some reason I don't get) we think that relative brain size matters more for intelligence than absolute brain size.
      >
      > > Conversely, human brains, or brains of any size for that matter, could have been exactly the same size but dominated by long-range connections rather than short range ones. In fact, in such a situation connectivity would actually be lower overall because long-range connections take up more space. But the mother's anatomy poses no obstacle to such a scenario. The fact that we don't see such a connectivity in any species is therefore plausibly a result of the fact that the encoding simply can't describe it easily.
      > >
      > >
      > Such a brain could exist, but because of the extra space taken up by the connections, there would have to be either fewer neurons, fewer connections (as you point out), or both. That certainly sounds like a cost to me! Moreover, longer connections are less reliable, cost more to build and maintain, and they are energetically more expensive due to signal drop along the length of the wire. So there are a whole host of costs associated with such a strategy, making me think it much, much more likely that these costs are the reason we don't see this type of brain, instead of it being difficult to encode. Evolution has produced amazing things, and it already produces many long-range connections in the brain, so I do not think the difficulty of encoding such a brain is what prevented it from existing.
      >
      > >
      > > I feel that you may not see what I'm saying about encoding here, because you speak about encoding as if it has similar effects to fitness pressure, but I think it's not the same. You say:
      > >
      > > "The reasons biases work is because they do bias search towards some areas and away from others: so I think both encoding biases and fitness penalties have similar effects in this regard."
      > >
      > > But I don't think that's true for encoding. The difference with encoding is that it is not pushing the search towards any particular area within the space it induces. Absent any kind of fitness function or selective pressure, encoding says nothing about which areas are accessible. Rather it simply says which types of phenotypes are overrepresented or underrepresented throughout the whole space of genotypes. In other words, even if an encoding is "biased" towards low connectivity, if you happen to get into an area of the space with high connectivity, the encoding does not have to push you out of that area (it could be a dense subspace full of high-connectivity structures). But fitness bias would have to push you out because all it does it push you out. It can't bend or change depending on where you go. Encoding can change with the times and has the wonderful additional potential for canalization, a bonus entirely absent from fitness pressure.
      > >
      > >
      > First of all, an encoding can actually prevent you from accessing a space. If I encode the length of all table legs in one number, than I have eliminated the possibility of a table having legs of different lengths. L-systems tend to produce such overly rigid biases (unless they are parameterized and/or made context-dependent). But more relevant to our discussion are biases that are strong likelihoods, but not strict edicts. Even these, though, in practice do mean that entire areas of the search space go unexplored. In our TEC paper, for example, the HyperNEAT generative encoding far outperformed the direct encoding, even though the fitness function was the exact same. Why? Because the direct encoding was biased towards an entirely different area of the search space. We know that there was a selection pressure for certain types of ANNs (namely, the ones HyperNEAT produced), and we know that the direct encoding can express those phenotypes (they actually do in HybrID), but evolution with the direct encoding did not do so because biases have a huge effect on the subset of the search space you visit. In fact, it is precisely because encodings have biases of large effect that we all care about generative encodings, no? All of these arguments are also supported by Hornby's comparison of L-systems to a direct encoding, including his maps of the types of phenotypes produced (there are huge differences between the two encodings). I'd argue that any comparison of direct and indirect encodings shows that these biases are not subtle, but grossly change the types of phenotypes explored, and for all practical purposes eliminate large swathes of the search space. For these reasons I think you'll get huge unintended consequences by playing around with encodings, and those consequences will not be overridden by the fitness function, because we've seen it happen time and time again. Note that I agree that you also have unintended consequences when playing with fitness functions.
      >
      > NB: I agree with you about canalization, which is one of many reasons I like generative encodings. :-)
      >
      > > The right level of modularity is likely on a delicate continuum - not entirely one way or another, and probably varies by species. You believe evolution can pay a kind of tax for going against the pressure towards low connectivity: "if a certain phenotype pays for its wiring by increasing fitness, it can add high-connectivity areas anywhere that they are useful." But in a delicate balancing act where the best option is likely some middle ground, that sounds too much like gambling and it's vulnerable to deception in cases where there is no immediate fitness benefit (whereas encoding is orthogonal to fitness). With encoding you don't have the play that game. Encoding can create its own tendency towards some middle ground and canalize that tendency over time.
      > >
      > If there is no fitness gradient toward such a bias toward intermediate connectivity, why would it evolve? Evolution can only think short term. I'll give you that an encoding bias might override the fitness gradient by making it impossible to follow, but I don't buy that evolution can magically figure out biases that are helpful in the long-term if the short-term fitness gradients all point in another direction. Or, at least, that's something we hope that evolution might do, but it's a controversial, unproven, and theoretically tricky issue that I certain don't think we can bank on happening until we understand it a lot more.
      >
      > > While you worry that modularity might "evolve away," the idea that it cannot evolve away to varying degrees sounds worse to me. Natural evolution is generally good about not keeping all its eggs in one basket - a trait may evolve away in some branches but not in others. But for you to make an all-out attempt to bar such a deviation from square one is making a lot of strong assumptions about what we want to see 1 billion years in the future.
      > >
      > >
      > I'd argue that your bias in the first replicator without any sustained fitness pressure would have zero effect on creatures a billion years later, especially in the case where there is an active fitness gradient by default away from modularity (which we know there is, since modularity never evolves without a bias or fitness pressure). Evolution would not keep any eggs in a basket with an active fitness penalty, at least not for a billion years. My 1-billion-years-later influence may thus be imperfect, but it at least exists!
      >
      > > So I'm still a fan of manipulating encoding over manipulating fitness. But I would not entirely despair on fitness because there will still be cases where there is no clear option for manipulating the encoding. But such scenarios are not ones we should be hoping for.
      > >
      > >
      > I do think you make some great arguments for encodings, but I cannot envision a case in which an initial bias only makes a huge difference in the long-term. That's true even if the bias is neutral with respect to fitness (because it would drift away), but seems a certainty to me if it relates to a bias that has an active fitness penalty. The whole point of canalization is that it figures out what produces fit offspring and generates that type of organism: if modular creatures are less fit in the short term, evolution will canalize away from modularity, not toward it. That said, as I mentioned at the beginning of this conversation, I do think that a sustained encoding bias of some sort is an interesting approach that could work, although it may be that all we have to do is provide a fitness cost and then the encoding will canalize in a way that produces such a sustained bias. :-)
      >
      > As always, an interesting conversation!
      > Best,
      > Jeff
      >
      >
      > > Best,
      > >
      > > ken
      > >
      > > --- In neat@yahoogroups.com, Jeff Clune wrote:
      > > >
      > > > Hello all,
      > > >
      > > > As Ken mentioned, we've discussed these issues in private. I'm going to include some of my comments from one of those email threads with slight modification, as I believe they summarize the views of Jean-Baptiste and I on the issues Ken raises. I'll then respond to a few individual comments by Ken afterwards.
      > > >
      > > > ---------------
      > > > Ken,
      > > >
      > > > It's great to hear your feedback on our paper. Thanks for sending it.
      > > >
      > > > First off, thanks for the kind words. We're very glad you liked the paper and think it is important.
      > > >
      > > > Regarding a selection pressure vs. an encoding bias. We're not convinced that an initial encoding bias is a good way to encourage properties that one wants in phenotypes throughout evolution, such as modularity. If there is any deceptiveness (or even neutrality) regarding modularity at any point during the run then the bias will disappear, and then for the rest of evolutionary time nothing will encourage modularity. We are more convinced of the power of mutational bias in the encoding (i.e. a constant encoding bias instead of just an initial encoding bias), and we think it would be interesting to investigate area. However, if the encoding bias is under selection, then you have the same issue where it might evolve away. Selective pressures are interesting because they are constant, so you're more likely to get what you want. That raises the point you mention about our pressure being too strong, such that evolution could not deviate when it would be beneficial not to have modularity. That might be a problem if the pressure is too strong, but it seems likely that in many cases the benefits in terms of performance for being non-modular will outweigh the cost. In other words, evolution can decide to pay the cost of non-modularity when it is useful (e.g. in your example of a hub of connections between modules).
      > > >
      > > > Regarding playing with an encoding being safer than playing with selection pressures. Our view is that both are very complicated and can have unintended consequences, so playing with one is just as bad as the other. I think our field is more familiar with unintended consequences of selective pressures just because we historically tend to play with them more (and make simple encodings), but it is also very hard to intuit the consequences of choices regarding biases in complex encodings. In your case the consequences are relatively intuitive, precisely because they are so minimally interventionist...but that is also why I think they are not strong enough to cause modularity except in cases (like retina) where all you have to do is initially place evolution in the right attractor basin.
      > > >
      > > > Regarding the resource hog waste of having a cost objective. I have to have a little fun here and point out the irony of the co-champion of novelty search worrying about the resources consumed by non-high-performing individuals! Hehe. As you've persuaded me, I'm more interested in an algorithm that is interesting or that works than spending a little computation inefficiently.
      > > >
      > > > I'd also like to point out an innovation we came up with to mitigate the problem of preventing evolution from exploring solutions that are contrary to one of the objectives. We recognized that the cost objective is ultimately less important than the performance objective. We wanted evolution to periodically ignore the cost objective to explore stepping stones that had higher connectivity. To do that, we invented a technique that involves "probabilistic pareto dominance", wherein secondary objectives (in this case cost) are factored into pareto dominance only a small percentage of the time. That won't solve the problem you mention if you have to take a long, many-multi-generational walk through high-connectivity areas of the search space, but it does allow quick forays into that terrain without any fitness penalty. This technique could be used for any cost (or other) objective, so it is not specific to connectivity costs.
      > > >
      > > > See below for a few specific responses to your comments. I should note that below this the thoughts are my own and Jean-Baptiste should not be blamed for any of them! (Feel free to blame him for things above this line…we went over that text together a while back). ;-)
      > > >
      > > > > More generally the issue is the usual problem of deception, which is compounded by anything you do with fitness. For example, in a complex search space, there is a reasonable chance that the stepping stone to a good low-connectivity solution is something with higher connectivity. By manipulating fitness, you are cutting out all chances of encountering such a deceptive stepping stone. But even if you don't believe that could be true, the single-mindedness of always favoring low-connectivity could deceive you from many parts of the search space that might be stepping stones to something worthwhile, relating to connection density or not.
      > > > >
      > > > >
      > > > True. But the same exact thing can be said for biases in the encoding: they prevent you from searching large areas of the search space. You may reply that it is only a bias, not a strict ban, but of course we know that in large search spaces biases hugely affect the landscape such that certain areas will practically never be visited.
      > > >
      > > > > On the other hand, manipulating the encoding is different because in effect it actually reorganizes the structure of the search space itself, which seems to me a more principled thing to do (if you can figure out a way to do it). Because the thing is, in that case, you do not need to worry about a permanent dead weight taking up some proportion of your population forever. Instead, while the encoding may *tend* to produce e.g. low-connectivity solutions, it can still escape that tendency without any penalty to fitness.
      > > > >
      > > > My instincts tell me that we create dead weight with encoding biases too. For example, an overly regular generative encoding (e.g. context free L-systems) is great if good solutions are perfectly regular, but if what is required is a mix of regularity and irregularity, then you spend your entire time producing only highly regular phenotypes that never wander into the appropriately irregular areas of the search space. Our IEEE TEC paper, for example, shows that HyperNEAT can spend thousands of generations spinning its wheels never generating solutions that HybrID could easily generate, demonstrating a "overly regular" dead weight associated with the biases of even the best known* generative encoding! Both fitness penalties and biases can cause you to focus your search in unproductive areas…which is ultimately what dead weight is.
      > > >
      > > > * in our opinion! :0)
      > > >
      > > > > Furthermore, in reality the best situation regarding modularity and connectivity is probably rather subtle, with most of the brain respecting the principle of low connectivity, but with a number of critical exceptions in key areas, such as major inter-module hubs. A sophisticated encoding can allow its bias to bend to make such nuanced exceptions (e.g. based on locations within a geometry), whereas a fitness penalty is a heavy hand and blunt instrument that cannot but help always to demand global and holistic subservience to dogmatic universals (unless you are willing to take a hit in fitness).
      > > > >
      > > > >
      > > > I think the last clause you offer is the key exception though. As I mentioned above, if a certain phenotype pays for its wiring by increasing fitness, it can add high-connectivity areas anywhere that they are useful (without even needing to carve out that area in geometric space, which is often a difficult task for CPPNs). Instead of being a blunt instrument, a fitness penalty can be quite subtle, because it can allow connection-by-connection exceptions if they produce fitness improvements, and do so without any search overhead.
      > > >
      > > > > An interesting question in nature (where our brains evolved modular structure) is whether its tendency towards low connectivity is a result of an aspect of fitness in the wild, or an aspect of encoding bias. I think there is a lot of room in this question for arguing either way, but my hunch is that the bias is mostly in the encoding. My logic is that I think the reason that the connectivity of the brain is so much lower than what it could be (e.g. it is a tiny fraction of everything-to-everything connectivity) is an artifact of physics rather than an artifact of fitness. It is simply physically impossible for a giant 100-billion-to-100-billion connectivity to fit in a head anything close to our size. And physical impossibility is in some sense a property of encoding. That is, mutations that could step from a low-connectivity brain to a high one are few and far between simply because of physical constraint. So high-connectivity structures are simply a very small part of the search space of brains in the physical universe. However, at the same time, you can still get long-range connections from time to time because there is no universal penalty for doing so, just a lower a priori probability of such mutations occurring.
      > > > >
      > > > >
      > > > Here I completely disagree with you. I see the force preventing the volume of neural connections from getting too large as a direct fitness cost, not an encoding bias. If mutations increase the size of the head, the baby and the mother are more likely to die in childbirth. Anthropologists have long known that evolution's desire to have larger and larger brains is the main reason why humans have such ridiculously high maternal and infant mortality "in the wild" (pre modern health care, and even post). Our encoding keeps producing such mutants, and it's death (via the physical constraints of the pelvis) that keep them from being kept around. The historical accident of birthing through the pelvis aside, there would still be fitness consequences for more vastly neural connections (e.g. neurons are metabolically expensive, neural connections require energy to build and maintain, and housing such large brains would create a large, clunky bobble head of a being that would be ungainly). It is possible that low connectivity is an encoding bias, but were that true I think it would look like some sort of growth rule that only grew a few connections per neuron (e.g. 10k). Evolution could have learned such a rule, and canalized that rule in a way that makes it unlikely to have mutations that produce orders of magnitude more neurons, but my guess is that if it has done so, it was because of the fitness costs associated with large numbers of neurons, not because of evolvability (or due to some historical accident). I do think it is interesting to study whether such biases exist in the encoding of neural growth rules, but even if they do exist I don't think that shows that a fitness cost was not the ultimate cause. Note: I recognize that you admit that this could be argued "either way"…are these the sorts of arguments you envisioned as being the other way?
      > > >
      > > > > In summary, the key difference between the alternatives is that with fitness you are saying "stay out of this part of the search space" whereas with encoding you are saying "this part of the search space is much smaller and hence less likely to encounter."
      > > > >
      > > > >
      > > > I don't precisely understand your latter clause, but I think I understand the spirit of it. I disagree, though. The reasons biases work is because they do bias search towards some areas and away from others: so I think both encoding biases and fitness penalties have similar effects in this regard.
      > > >
      > > > > So, my speculation is that if you want to bias the search in highly complex domains, the best way is through the encoding. Fitness is a nasty quagmire that is deceptively tempting to manipulate, but never plays by the rules you wish it would. Of course, these are merely my own unproven intuitions and their veracity remains to be demonstrated. But at least it's something to think about.
      > > > >
      > > > >
      > > >
      > > > Thanks again for your feedback. As always, I appreciate your input and enjoy discussing these fascinating subjects.
      > > >
      > > > I want to end by clarifying that I agree that there are positives and negatives to both approaches. I do not see it as such an obvious choice between the two as you do. As I mentioned up top: I definitely don't think initial encoding biases that selection can get rid of will get us very far. For simple problems they will work, but for any challenging problem the initial bias will have long disappeared by the time it will matter. What we need is some constant force encouraging search to take promising paths. A fitness cost is one way to do that, but a *constant* (or, at least, *periodic*) encoding bias could do that just as well, and perhaps better.
      > > >
      > > >
      > > > Best regards,
      > > > Jeff Clune
      > > >
      > > > Assistant Professor
      > > > Computer Science
      > > > University of Wyoming
      > > > jeffclune@
      > > > jeffclune.com
      > > >
      > > >
      > > > > Best,
      > > > >
      > > > > ken
      > > > >
      > > > > --- In neat@yahoogroups.com, Alexandre Devert wrote:
      > > > > >
      > > > > > Hi,
      > > > > >
      > > > > > Â Simple, clean experiment, with sharp results, congrats on that, definitely
      > > > > > a step forward ! Of course, it begs for more questions. I would love to hear
      > > > > > you on such (fairly open) questions
      > > > > >
      > > > > > Â Â 1) Do you think that selection pressure for low connectivity is sufficient in
      > > > > > itself to evolve large coherent networks, or is it just a piece of the puzzle ?
      > > > > > Â Â 2) Do you see your work as an indication that any approach biased to low
      > > > > > connectivity would reproduce the result ? Or does the way you guys enforced
      > > > > > this bias matters ?
      > > > > >
      > > > > > To meÂ
      > > > > > 1) => Part of the puzzle. Should see how well it scales for increasingly
      > > > > > complex task, when the connection graph gets bigger. A randomizedÂ
      > > > > > search process on large graph sounds not so efficient, need something to guide it.
      > > > > > I advocate construction process that have a feedback from what the neuronÂ
      > > > > > network is computing. Don't know how to do it without creepling computational
      > > > > > cost tho...
      > > > > > 2) => I guess that the bias alone is enough, the way to introduce it might
      > > > > > not be such a big deal.Â
      > > > > >
      > > > > > Again, great work, very helpful contribution :)
      > > > > >
      > > > > > Alex
      > > > > > Â
      > > > > > Dr. Devert Alexandre
      > > > > > Researcher at the Nature Inspired Computation and Applications Laboratory (NICAL)
      > > > > > Lecturer at School Of Software Engineering of USTC
      > > > > > ----------------------------------------------------
      > > > > > Homepage :Â http://www.marmakoide.org
      > > > > > ----------------------------------------------------
      > > > > > 166 Renai Road, Dushu Lake Higher Education Town
      > > > > > Suzhou Industrial Park,
      > > > > > Suzhou, Jiangsu, People's Republic of China
      > > > > >
      > > > > >
      > > > > > ________________________________
      > > > > > From: Jeff Clune
      > > > > > To: neat users group group
      > > > > > Cc: Jean-Baptiste Mouret ; Hod Lipson
      > > > > > Sent: Thursday, February 7, 2013 1:57 AM
      > > > > > Subject: [neat] New paper on why modules evolve, and how to evolve modular artificial neural networks
      > > > > >
      > > > > >
      > > > > > Â
      > > > > > Hello all,
      > > > > >
      > > > > > I'm extremely pleased to announce a new paper on a subject that many--including myself--think is critical to making significant progress in our field: the evolution of modularity.Â
      > > > > >
      > > > > > Jean-Baptiste Mouret, Hod Lipson and I have a new paper thatÂ
      > > > > >
      > > > > > 1) sheds light on why modularity may evolve in biological networks (e.g. neural, genetic, metabolic, protein-protein, etc.)
      > > > > >
      > > > > > 2) provides a simple technique for evolving neural networks that are modular and have increased evolvability, in that they adapt faster to new environments. The modules that formed solved subproblems in the domain.Â
      > > > > > Cite: Clune J, Mouret J-B, Lipson H (2013) The evolutionary origins of modularity. Proceedings of the Royal Society B. 280: 20122863. http://dx.doi.org/10.1098/rspb.2012.2863%c3%82 (pdf)
      > > > > >
      > > > > > Abstract: A central biological question is how natural organisms are so evolvable (capable of quickly adapting to new environments). A key driver of evolvability is the widespread modularity of biological networksâ€"their organization as functional, sparsely connected subunitsâ€"but there is no consensus regarding why modularity itself evolved. Although most hypotheses assume indirect selection for evolvability, here we demonstrate that the ubiquitous, direct selection pressure to reduce the cost of connections between network nodes causes the emergence of modular networks. Computational evolution experiments with selection pressures to maximize network performance and minimize connection costs yield networks that are significantly more modular and more evolvable than control experiments that only select for performance. These results will catalyse research in numerous disciplines, such as neuroscience and genetics, and enhance our ability to harness
      > > > > > evolution for engineering purposes.
      > > > > >
      > > > > > Video:Â http://www.youtube.com/watch?feature=player_embedded&v=SG4_aW8LMng
      > > > > >
      > > > > > There has been some nice coverage of this work in the popular press, in case you are interested:
      > > > > > National Geographic:Â http://phenomena.nationalgeographic.com/2013/01/30/the-parts-of-life/MIT's Technology Review:Â http://www.technologyreview.com/view/428504/computer-scientists-reproduce-the-evolution-of-evolvability/%c3%82 Fast Company:Â http://www.fastcompany.com/3005313/evolved-brains-robots-creep-closer-animal-learningCornell Chronicle:Â http://www.news.cornell.edu/stories/Jan13/modNetwork.htmlScienceDaily:%c3%82 http://www.sciencedaily.com/releases/2013/01/130130082300.htm
      > > > > >
      > > > > > Please let me know what you think and if you have any questions. I hope this work will help our field move forward!
      > > > > >
      > > > > >
      > > > > >
      > > > > >
      > > > > > Best regards,
      > > > > > Jeff Clune
      > > > > >
      > > > > > Assistant Professor
      > > > > > Computer Science
      > > > > > University of Wyoming
      > > > > > jclune@
      > > > > > jeffclune.com
      > > > > >
      > > > >
      > > > >
      > > > >
      > > >
      > >
      > >
      >
    • Ken
      Hi Jeff, I don t mean to drag on the conversation too long but it s getting to some really interesting issues, some of which go beyond the one topic of
      Message 2 of 30 , Feb 22, 2013
      • 0 Attachment
        Hi Jeff,  I don't mean to drag on the conversation too long but it's getting to some really interesting issues, some of which go beyond the one topic of modularity, so I feel it's still productive to respond on some of your points.  To keep the thread readable I didn't keep every single item from before.  (My responses are the ones with the least indent.)

        Nature doesn't have anything analogous either, which means there is at least some evidence that the "fitness bias" analogy with nature is not lining up perfectly. You might point to the continuing existence of single-celled organisms as something similar to the perpetual dead-weight in this formulation, but they aren't really analogous because single-celled organisms are functional - they retain the ability to make copies of themselves and continue to evolve in their own right - while the low-connectivity deadweight maintains no capability whatsoever. On the other hand, suspiciously, as in nature, nothing similar to such a deadweight niche is perpetuated by a biased encoding.

        That's also true, but that fault does not lie with the fitness cost concept, it lies with the fact that multi-objective algorithms, which are the cause of the dead weight, do not perfectly analogize to nature. They're just better than a weighted sum for other reasons, but the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue.  

        Doesn't it seem a little strange that the price we have to pay to obtain modular structure is to maintain a perpetual dead pool of genetic junk? Note that it doesn't suggest that such a system won't work in some cases, but it's inelegant enough to raise questions about the best realization of the concept..


        All I think that calls into question is the optimality of multi-objective algorithms when you don't want the extreme of one objective. But that problem almost always occurs in multi-objective algorithms, so your really indicting the whole field of MOEA instead of our approach of using a fitness penalty instead of a biased encoding, no?

        No I don't think I'm indicting MOEAs in general.  The problem here is not some inherent problem with MOEAs, but that MOEAs were not designed for the purpose for which you have borrowed them.   MOEAs are designed to return a Pareto front given a number of objective trade-offs.  They are doing that perfectly well in your experiment.  It's just that your interest is not perfectly aligned with that design:  Your interest is in the "best tradeoff" (which is inherently slippery to formalize and not formalized by a traditional  MOEA), while some trade-offs that MOEAs are by design made to preserve and elevate (such as total dominance on low connectivity) are of almost no interest in this domain, as you've agreed.  So my general critique of manipulating the fitness function has nothing to do with MOEAs specifically.

        Rather, what we are observing is simply the struggle to find some kind of expression of fitness that aligns with what you actually want to see (and thereby incentivizes following the right path through the search space).  And the fact that you settled on MOEAs as the best option, and that they do not align all that well, just illustrates how nasty this little problem of tweaking fitness really is. 

        You noted that MOEAs are "just better than a weighted sum for other reasons," but that's exactly the problem.  Those "other reasons" are indeed unrelated to the problem you are trying to solve here.  That is, MOEAs are a blunt kludge in this particular context.  But although you defend fitness in general with the response that "the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue," the problem there is that you once again end up with similarly unnatural and incongruous implications:

        If you make fitness a weighted sum and one component of that weighted sum is "low connectivity," then there will *still* be a special eternally protected pocket for non-functional low-connectivity structure.  The reason is that if there is any progress on actually solving the problem (aside from connectivity), then those networks that are moving towards solving it will be more connected than the lowest possible connectivity.  Very often, because of mutation, some of these more connected networks will break, leading to an inevitable (and unavoidable) subpopulation  of higher-connectivity networks that are broken and nonfunctional.  Because of this inevitable subset of failures in every generation, a great strategy for some of the population is to stay as minimally connected as possible while ignoring functionality (just as in the MOEA) because they can be just barely  "good enough" to keep perpetuating merely by their connectivity fitness bonus versus those who are more connected and also nonfunctional.

        Now you may then reason that there is some more complicated way to counteract this problem.  For example, you may say, well, we just need to be more strict about selection, so amp up selection pressure by blocking more of the population from reproducing.  But you have no way to know a priori what that threshold should be, so you are risking breaking evolution in other ways (such as becoming too convergent) by wiping out diversity.  In fact, early in evolution, when most structure is nonfunctional or less functional, that could short circuit the  whole process. 

        But I want to emphasize that my point here has nothing to do with trying to figure out the right "trick" to get fitness to actually align properly with what we want to see, or with natural evolution of modularity.  If you notice, when you begin to talk about using probabilistic Pareto fronts or something like that, you are just playing again the same "let's tweak things around hoping to get it right" game.  And my point here is that that game is ultimately a game of futility.  Sure, in this very simply task (relative to finding a natural brain), virtually anything will work even if it is radically out of whack with nature, so you can convince yourself that this problem is possible to reconcile.  You can convince yourself that with enough tweaking you'll just write down the magic fitness/MOEA recipe that equals finding what you want.  But the search space is so incredibly complex that such a dream is virtually impossible.  Because if you think about it, as you make one tweak after another, as you fix one unintended consequence with yet another trick, what you are ultimately doing is describing the path through the search space itself.  In other words, the logical extension of such a process is simply to identify all the stepping stones from all possible random starting points to the objective and give higher fitness for each step in the chain, a task akin to applied omnipotence.  You might as well just build the solution by hand in that case, because you would know all the stepping stones to the solution anyway.  So while these fitness tricks may work for now, while we play in modest playgrounds, there are big warning signs looming in the future.


        I feel that you may not see what I'm saying about encoding here, because you speak about encoding as if it has similar effects to fitness pressure, but I think it's not the same. You say:

        "The reasons biases work is because they do bias search towards some areas and away from others: so I think both encoding biases and fitness penalties have similar effects in this regard."

        But I don't think that's true for encoding. The difference with encoding is that it is not pushing the search towards any particular area within the space it induces. Absent any kind of fitness function or selective pressure, encoding says nothing about which areas are accessible. Rather it simply says which types of phenotypes are overrepresented or underrepresented throughout the whole space of genotypes. In other words, even if an encoding is "biased" towards low connectivity, if you happen to get into an area of the space with high connectivity, the encoding does not have to push you out of that area (it could be a dense subspace full of high-connectivity structures). But fitness bias would have to push you out because all it does it push you out. It can't bend or change depending on where you go. Encoding can change with the times and has the wonderful additional potential for canalization, a bonus entirely absent from fitness pressure.

        First of all, an encoding can actually prevent you from accessing a space. If I encode the length of all table legs in one number, than I have eliminated the possibility of a table having legs of different lengths. L-systems tend to produce such overly rigid biases (unless they are parameterized and/or made context-dependent). But more relevant to our discussion are biases that are strong likelihoods, but not strict edicts. Even these, though, in practice do mean that entire areas of the search space go unexplored. In our TEC paper, for example, the HyperNEAT generative encoding far outperformed the direct encoding, even though the fitness function was the exact same. Why? Because the direct encoding was biased towards an entirely different area of the search space. We know that there was a selection pressure for certain types of ANNs (namely, the ones HyperNEAT produced), and we know that the direct encoding can express those phenotypes (they actually do in HybrID), but evolution with the direct encoding did not do so because biases have a huge effect on the subset of the search space you visit. In fact, it is precisely because encodings have biases of large effect that we all care about generative encodings, no? All of these arguments are also supported by Hornby's comparison of L-systems to a direct encoding, including his maps of the types of phenotypes produced (there are huge differences between the two encodings). I'd argue that any comparison of direct and indirect encodings shows that these biases are not subtle, but grossly change the types of phenotypes explored, and for all practical purposes eliminate large swathes of the search space. For these reasons I think you'll get huge unintended consequences by playing around with encodings, and those consequences will not be overridden by the fitness function, because we've seen it happen time and time again. Note that I agree that you also have unintended consequences when playing with fitness functions. 

        NB: I agree with you about canalization, which is one of many reasons I like generative encodings. :-)

        We are in agreement that encodings matter.  Of course, that is part of why I'm saying they're important.  But what I'm really saying is that encodings are far better suited than fitness for the long haul, i.e. for doing something interesting over millions of generations, largely *because* a good encoding can canalize.  The trick behind radically successful long-term evolution, in my view, is to keep your options open.  You don't want to say X is clearly better than Y.  What you want to say is, if X and Y are fundamentally different, I want to check both X and Y and see where each of them lead and follow both branches if both are interesting.  But furthermore, if some property of X proves helpful in leading it to all kinds of cool stuff (or the same for Y), then let's start preserving (i.e. canalizing) that property THEN, when we observe its potential, rather than a priori from day one when we would have to be an omnipotent being to anticipate everything that might be useful.  And only encoding offers that possibility.  Fitness is all about making choices about priorities long before you have any idea what the options are.  Encoding with canalization is about giving evolution the choice to find the level that's right for it.  There is nothing analogous to canalization in fitness.

        Canalization is an intriguing issue that is not fully understood, but it does happen, clearly in nature, and even in CPPNs.  For example, one cool experiment I tried informally on Picbreeder was to keep regenerating new populations from the same starting image over and over again from both (1) a highly-evolved image and (2) a low-generation image.  Not surprisingly, the offspring of the highly-evolved image were significantly more consistent (based on my subjective perception) than of the less-evolved one.  That's canalization in action, even in Picbreeder.  By the way, applying an objective fitness function will undermine the potential for canalization (with CPPNs or anything else) because of the tendency of objectives to wreck the representation.  So fitness here has a huge disadvantage as a tool for manipulation:  Not only does it offer no mechanism remotely comparable to canalization, but it actually thwarts canalization in the encoding even if the encoding can do it.  It's like you keep whipping evolution for losing modularity and it keeps paying the price without every really getting the idea fundamentally (i.e. through the encoding).

        So this is why we want something like LEO where modularity *can* evolve away to different degrees:  We want to make no a priori assumptions about what must be correct in all generations, and instead allow evolution to try out everything.  And those things that lead to more potential over time will become canalized, you can be confident, and therein evolution works its magic. 

        So while you are saying regarding the human head size that DNA perhaps *can* encode brains with high connectivity but that fitness is preventing it, I would say that DNA *in general* perhaps can do that, but by now, long into the human lineage, these pseudo-modular structures are so deeply canalized that breaking out of that canal for the encoding would require herculean effort.  But that's beauty of encoding.

        The right level of modularity is likely on a delicate continuum - not entirely one way or another, and probably varies by species. You believe evolution can pay a kind of tax for going against the pressure towards low connectivity: "if a certain phenotype pays for its wiring by increasing fitness, it can add high-connectivity areas anywhere that they are useful." But in a delicate balancing act where the best option is likely some middle ground, that sounds too much like gambling and it's vulnerable to deception in cases where there is no immediate fitness benefit (whereas encoding is orthogonal to fitness). With encoding you don't have the play that game. Encoding can create its own tendency towards some middle ground and canalize that tendency over time.

        If there is no fitness gradient toward such a bias toward intermediate connectivity, why would it evolve? Evolution can only think short term. I'll give you that an encoding bias might override the fitness gradient by making it impossible to follow, but I don't buy that evolution can magically figure out biases that are helpful in the long-term if the short-term fitness gradients all point in another direction. Or, at least, that's something we hope that evolution might do, but it's a controversial, unproven, and theoretically tricky issue that I certain don't think we can bank on happening until we understand it a lot more. 

        I think the problem is that you are thinking of evolution as "thinking" short term or long term, and that you are hoping that some "gradient" will somehow describe for us the right path over all these disparate timescales.   Evolution indeed cannot think ahead.  Its job when it thrives is to try many things and see where they go.  Ad hoc a priori gradients will only get in the way of that.  Give the encoding the ability to support such diversified exploration and canalize the best of it, and evolution need not be thinking at all. 

        While you worry that modularity might "evolve away," the idea that it cannot evolve away to varying degrees sounds worse to me. Natural evolution is generally good about not keeping all its eggs in one basket - a trait may evolve away in some branches but not in others. But for you to make an all-out attempt to bar such a deviation from square one is making a lot of strong assumptions about what we want to see 1 billion years in the future.

        I'd argue that your bias in the first replicator without any sustained fitness pressure would have zero effect on creatures a billion years later, especially in the case where there is an active fitness gradient by default away from modularity (which we know there is, since modularity never evolves without a bias or fitness pressure). Evolution would not keep any eggs in a basket with an active fitness penalty, at least not for a billion years. My 1-billion-years-later influence may thus be imperfect, but it at least exists!

        That's why I don't like the word bias as much when it comes to encoding.  I'd think of it more as an option.  Evolution can try everything - preserve it in some lineages, to a less degree in others, and not at all in others still.  The ability to commit to subtlety is the magic here.  That's the smart strategy - keep your options open but also have the chance to commit to an option when it's working for you.  A single blunt bias imposed for a billion years (which is about closing off options) does not sound like a smart strategy in a search space more astronomical than imagination itself.

        So I'm still a fan of manipulating encoding over manipulating fitness. But I would not entirely despair on fitness because there will still be cases where there is no clear option for manipulating the encoding. But such scenarios are not ones we should be hoping for.

        I do think you make some great arguments for encodings, but I cannot envision a case in which an initial bias only makes a huge difference in the long-term. That's true even if the bias is neutral with respect to fitness (because it would drift away), but seems a certainty to me if it relates to a bias that has an active fitness penalty. The whole point of canalization is that it figures out what produces fit offspring and generates that type of organism: if modular creatures are less fit in the short term, evolution will canalize away from modularity, not toward it. That said, as I mentioned at the beginning of this conversation, I do think that a sustained encoding bias of some sort is an interesting approach that could work, although it may be that all we have to do is provide a fitness cost and then the encoding will canalize in a way that produces such a sustained bias. :-)
        I think your view of evolution here is too linear (as opposed to branching), which is why you tend to worry about things like what is good in the short term or long term.  But the idea that you can impose these blunt constraints on evolution for eternity fits with a linear perspective, so it makes sense if that's how you view it, as a kind of step-by-step succession along a single chain.  But I think evolution in nature is not walking this single tightrope towards some kind of near-optimal ideal, where all our effort needs to focus on not falling off the tightrope.   That is a very objective view, and I think the evidence is pointing more and more towards why that kind of perspective has been an impediment for us.  But I agree the conversation has been great and hope I made my points respectfully.   I realize we're wading into controversy and that reasonable minds can disagree here.

        Best,

        ken



      • Ken
        Hi Stephane, I want to acknowledge that the work on multiobjective optimization is important, and the contributions of yourself, JB, and others are very
        Message 3 of 30 , Feb 22, 2013
        • 0 Attachment
          Hi Stephane,

          I want to acknowledge that the work on multiobjective optimization is important, and the contributions of yourself, JB, and others are very thought-provoking, as well as practical. One of the important caveats to my debate with Jeff is that I am advocating using the encoding to encourage certain properties when that is possible, but that often it will not be possible because the relationship between the encoding and what we want will be too difficult to comprehend. So there will be many cases where fitness-based strategies such as MOEAs will be a viable choice. As you know, my own group has also worked with MOEAs in a number of combinations in recent years (such as NS + local competition) as well, inspired in part by your own demonstrations of their potential. We also have a multiobjective version of NEAT, which replaces the traditional speciation with a genetic diversity objective (with the idea that you can then more easily combine NEAT with NSGA II). Your result in ball collecting are also impressive.

          So I am by no means against MOEAs and I'm afraid my argument with Jeff is coming off that way. My argument is rather that encoding is often going to be the most natural and least ad hoc place to encourage certain properties if you can figure out how to do it through the encoding. While you are right that MOEAs are convenient in the sense they work with any encoding, I don't think in the end evolution is about convenience, so we'll have to grapple with encoding. As Jeff points out, encoding can make a big difference (like L-systems vs. CPPNs vs. direct encodings). But I don't think you have much disagreement with that either.

          Often when I think about evolution I'm thinking about long-term (like millions of generations), open-ended situations, or natural evolution. I'm thinking about evolving amazing things that rival what we see in nature. I'm thinking, okay so we evolved a toy problem, but how far can we extrapolate the lessons to evolving a brain. And I think there's a lot of danger there that we extrapolate too much from lessons that are learned in astronomically simpler contexts, especially when it comes to fitness pressures. So that's why I think it's important to be open-minded to the possibility that what looks important in the short run is a red herring in the long run. But that doesn't mean I'm in any sense I'm dismissing MOEAs altogether.

          Best,

          ken



          --- In neat@yahoogroups.com, "stephane.doncieux" <stephane.doncieux@...> wrote:
          >
          >
          >
          > Hi,
          >
          > Congratulations to JB, Jeff and Hod for their great work on the
          > evolution of modularity !
          >
          > The length and multiplicity of the comments undoubtedly show the value
          > of this work. I would like to comment on the debate between encoding bias and fitness manipulations that this paper has provoked in the mailing list (a debate that actually goes beyond the connection cost objective for modularity).
          >
          > After having spent a long time on encodings, I find it fruitful to
          > work on selection pressures. We have actually worked on the definition
          > of such fitness pressures and have proposed objectives to promote
          > diversity (Behavioral diversity [1]), generalization [2], behavior
          > consistency [3,4] or to cross the reality gap with robots [5]. I
          > don't want to say that it is the only way to solve the problems that
          > these objectives are designed for, nor that it is the best way to do
          > it, but I will just mention the advantages I find with such an
          > approach relative to work on the encoding.
          >
          > The great advantage of the works on selection pressures is that it is
          > highly localized. The contribution is limited to the definition of an
          > objective that is then easy to integrate to a third party work than
          > when it comes with its own specific selection algorithm and
          > encoding. Such work does not highly depend on an encoding, nor on a
          > particular selection algorithm (as long as it is a multi-objective
          > algorithm). It is trivial to make comparisons with-without a
          > particular objective and all these works are, at least from a
          > technical point of view, compatible. You can then use, besides a
          > goal-oriented objective, the connection cost objective for modularity together with a behavioral diversity or novelty objective to increase the exploration. What is even more interesting in my opinion, is that
          > anyone interested in optimizing complex neural networks or graphs and
          > that may have his own encoding for whatever reasons (to be
          > biologically plausible for instance) can use these objectives in his
          > own work with very few, if any, technical modifications. It represents
          > then a lower investment and makes such kind of work more prone to be
          > used outside of our own field, it is at least my opinion on the
          > subject. Likewise, all the works mentionned above rely on NSGA-II,
          > i.e. a classic multi-objective evolutioanry algorithm. Any
          > new and more efficient MOEA can by used without any particular
          > technical problem.
          >
          > To come back to the performance issue, in all of our experiments,
          > using a simple direct encoding with two objectives
          > (fitness+behavioral diversity) gave better results than NEAT, no
          > matter how we define behavioral diversity [1]. What is nice with these
          > results, is that it is not incompatible with NEAT. A
          > multi-objective version of NEAT can exploit the benefits gained with
          > behavioral diversity (it has actually been proposed in a GECCO paper
          > by Moriguchi and Honiden). A quite surprising result for us is that a
          > simple direct encoding can solve a complex sequential tasks (ball
          > collecting task) with a discrete and simple fitness function (a number
          > of collected balls) together with a simple behavioral diversity
          > objective (see a video here: http://youtu.be/Trj0_A1ZfNo).
          >
          > Of course, there are limitations. If you use more than three
          > objectives, for instance, the performance of multi-objective
          > algorithms is known to be very low. This is a limitation that people
          > from the multi-objective optimization community are aware of. They
          > work on it and many-objectives optimization algorithms will probably
          > be released in a near future (some have already been proposed
          > actually). Likewise, as pointed out by Jeff, the dead weights
          > mentionned by Ken are really a consequence of current multi-objective
          > evolutionary algorithms inner workings. Several adaptations can be
          > proposed, as the "probabilistic pareto dominance" developped by Jeff
          > and Jean-Baptiste or any other preference or thresholding system. It
          > is not an inherant limitation of these approaches, just a proof that
          > there is still some work to do in multi-objective optimization
          > field. It should be noted that this is not a problem of our community
          > only. Multi-objective optimization is a very dynamic field of research
          > so we can expect solutions to these problems in a forthcoming future.
          >
          > So to conclude, I agree that it is important to work on the encoding,
          > but my point is that it is also worth to work on selection pressures. The nice thing is
          > that the works on both aspects are quite complementary.
          >
          >
          > Best,
          >
          > Stephane Doncieux
          >
          > [1] Mouret, J.-B. and Doncieux, S. (2012). Encouraging Behavioral Diversity in Evolutionary Robotics: an Empirical Study.
          > Evolutionary Computation. Vol 20 No 1 Pages 91-133.
          >
          > [2] Pinville, T. and Koos, S. and Mouret, J-B. and Doncieux,
          > S. (2011). How to Promote Generalisation in Evolutionary Robotics: the
          > ProGAb Approach. GECCO'11. Pages 259--266.
          >
          > [3] Ollion, Charles and Doncieux, Stéphane (2012). Towards Behavioral Consistency in Neuroevolution.
          > From Animals to Animats: Proceedings of the 12th International
          > Conference on Adaptive Behaviour (SAB 2012), Springer, publisher.
          >
          > [4] Ollion, C. and Pinville, T. and Doncieux, S. (2012). With a little
          > help from selection pressures: evolution of memory in robot
          > controllers. Proc. Alife XIII.
          >
          > [5] Koos, S. and Mouret, J.-B. and Doncieux, S. (2013). The
          > Transferability Approach: Crossing the Reality Gap in Evolutionary
          > Robotics. IEEE Transactions on Evolutionary Computation. Vol 17 No 1 Pages 122 - 145 .
          >
          >
          >
          > --- In neat@yahoogroups.com, Jeff Clune <jclune@> wrote:
          > >
          > > Hello Ken,
          > >
          > > Please see my inlined comments.
          > >
          > > >
          > > > Hi Jeff, I wanted to follow up on the "dead weight" concept I brought up, because I feel I may not have made it entirely clear what that is. In a multiobjective formulation, if one of the objectives is low connectivity, then you can dominate on that objective by having extremely low connectivity and no other saving grace whatsoever. In other words we are talking about completely nonfunctional and essentially inert garbage that does well on that one objective by effectively being brain dead. In effect, you have created a special "niche" for brain-dead networks with radically low connectivity.
          > > >
          > > >
          > > That's true. Of course, you could choose to have a cutoff to minimize this problem. For example, you could not give credit to any orgs that do not have a path from inputs to outputs, or that have a total connectivity below X, or some other solution (including a cutoff that changes over time).
          > >
          > > > Perhaps early on this niche is a fruitful point of departure for better places, but the problem is that this niche will never go away. You will always have this protective pocket for brain-dead low-connectivity networks for as long as evolution runs. In fact it's a very attractive niche because it's so easy - you don't have to worry about performance and simply need to keep your structure down. So these types of inert blobs will be around forever.
          > > >
          > > > You suggest that my concern about this "unfit" niche is not consistent with my support of novelty search, but novelty search is not really analogous. Novelty search is about constantly searching for *new* departure points. Your dead-weight niche is about keeping around the same bottom-level junk forever. Novelty search would quickly tire of such a black hole and abandon it (it doesn't stay novel). But multiobjective search will embrace it for eternity.
          > > >
          > > >
          > > That's true, but on many complex/multi-dimensional tasks there are often infinitely many ways to be uninteresting, yet novel. For example, in your unenclosed map in the NS journal paper, only a few runs solve the problem (the same as random search), likely because of the permanent dead-weight of an infinite number of places to end up that are far away from the target.
          > >
          > > > Nature doesn't have anything analogous either, which means there is at least some evidence that the "fitness bias" analogy with nature is not lining up perfectly. You might point to the continuing existence of single-celled organisms as something similar to the perpetual dead-weight in this formulation, but they aren't really analogous because single-celled organisms are functional - they retain the ability to make copies of themselves and continue to evolve in their own right - while the low-connectivity deadweight maintains no capability whatsoever. On the other hand, suspiciously, as in nature, nothing similar to such a deadweight niche is perpetuated by a biased encoding.
          > > >
          > > >
          > > That's also true, but that fault does not lie with the fitness cost concept, it lies with the fact that multi-objective algorithms, which are the cause of the dead weight, do not perfectly analogize to nature. They're just better than a weighted sum for other reasons, but the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue.
          > >
          > > > Doesn't it seem a little strange that the price we have to pay to obtain modular structure is to maintain a perpetual dead pool of genetic junk? Note that it doesn't suggest that such a system won't work in some cases, but it's inelegant enough to raise questions about the best realization of the concept..
          > > >
          > >
          > > All I think that calls into question is the optimality of multi-objective algorithms when you don't want the extreme of one objective. But that problem almost always occurs in multi-objective algorithms, so your really indicting the whole field of MOEA instead of our approach of using a fitness penalty instead of a biased encoding, no?
          > >
          > > >
          > > > Also on the analogy with nature, while your argument for fitness pressure in nature based on the size of the head is logically possible (and the kind of argument that is probably attractive to a lot of people), it's only speculative. In fact what little evidence there is on such a speculative issue (i.e. whether a baby's head might fit through a mother's pelvis in some alternate evolutionary path) doesn't support the idea that size is the main issue here. After all, whale brains are far bigger than human brains and I'm sure they are also dominated by local connectivity.
          > > >
          > > Yes, but they float in water! You couldn't have a creature with that big of a brain on land (I speculate, not being an expert in these things). Interesting side note: I believe whale brains are pretty tiny as a fraction of body size compared to primates, and (for some reason I don't get) we think that relative brain size matters more for intelligence than absolute brain size.
          > >
          > > > Conversely, human brains, or brains of any size for that matter, could have been exactly the same size but dominated by long-range connections rather than short range ones. In fact, in such a situation connectivity would actually be lower overall because long-range connections take up more space. But the mother's anatomy poses no obstacle to such a scenario. The fact that we don't see such a connectivity in any species is therefore plausibly a result of the fact that the encoding simply can't describe it easily.
          > > >
          > > >
          > > Such a brain could exist, but because of the extra space taken up by the connections, there would have to be either fewer neurons, fewer connections (as you point out), or both. That certainly sounds like a cost to me! Moreover, longer connections are less reliable, cost more to build and maintain, and they are energetically more expensive due to signal drop along the length of the wire. So there are a whole host of costs associated with such a strategy, making me think it much, much more likely that these costs are the reason we don't see this type of brain, instead of it being difficult to encode. Evolution has produced amazing things, and it already produces many long-range connections in the brain, so I do not think the difficulty of encoding such a brain is what prevented it from existing.
          > >
          > > >
          > > > I feel that you may not see what I'm saying about encoding here, because you speak about encoding as if it has similar effects to fitness pressure, but I think it's not the same. You say:
          > > >
          > > > "The reasons biases work is because they do bias search towards some areas and away from others: so I think both encoding biases and fitness penalties have similar effects in this regard."
          > > >
          > > > But I don't think that's true for encoding. The difference with encoding is that it is not pushing the search towards any particular area within the space it induces. Absent any kind of fitness function or selective pressure, encoding says nothing about which areas are accessible. Rather it simply says which types of phenotypes are overrepresented or underrepresented throughout the whole space of genotypes. In other words, even if an encoding is "biased" towards low connectivity, if you happen to get into an area of the space with high connectivity, the encoding does not have to push you out of that area (it could be a dense subspace full of high-connectivity structures). But fitness bias would have to push you out because all it does it push you out. It can't bend or change depending on where you go. Encoding can change with the times and has the wonderful additional potential for canalization, a bonus entirely absent from fitness pressure.
          > > >
          > > >
          > > First of all, an encoding can actually prevent you from accessing a space. If I encode the length of all table legs in one number, than I have eliminated the possibility of a table having legs of different lengths. L-systems tend to produce such overly rigid biases (unless they are parameterized and/or made context-dependent). But more relevant to our discussion are biases that are strong likelihoods, but not strict edicts. Even these, though, in practice do mean that entire areas of the search space go unexplored. In our TEC paper, for example, the HyperNEAT generative encoding far outperformed the direct encoding, even though the fitness function was the exact same. Why? Because the direct encoding was biased towards an entirely different area of the search space. We know that there was a selection pressure for certain types of ANNs (namely, the ones HyperNEAT produced), and we know that the direct encoding can express those phenotypes (they actually do in HybrID), but evolution with the direct encoding did not do so because biases have a huge effect on the subset of the search space you visit. In fact, it is precisely because encodings have biases of large effect that we all care about generative encodings, no? All of these arguments are also supported by Hornby's comparison of L-systems to a direct encoding, including his maps of the types of phenotypes produced (there are huge differences between the two encodings). I'd argue that any comparison of direct and indirect encodings shows that these biases are not subtle, but grossly change the types of phenotypes explored, and for all practical purposes eliminate large swathes of the search space. For these reasons I think you'll get huge unintended consequences by playing around with encodings, and those consequences will not be overridden by the fitness function, because we've seen it happen time and time again. Note that I agree that you also have unintended consequences when playing with fitness functions.
          > >
          > > NB: I agree with you about canalization, which is one of many reasons I like generative encodings. :-)
          > >
          > > > The right level of modularity is likely on a delicate continuum - not entirely one way or another, and probably varies by species. You believe evolution can pay a kind of tax for going against the pressure towards low connectivity: "if a certain phenotype pays for its wiring by increasing fitness, it can add high-connectivity areas anywhere that they are useful." But in a delicate balancing act where the best option is likely some middle ground, that sounds too much like gambling and it's vulnerable to deception in cases where there is no immediate fitness benefit (whereas encoding is orthogonal to fitness). With encoding you don't have the play that game. Encoding can create its own tendency towards some middle ground and canalize that tendency over time.
          > > >
          > > If there is no fitness gradient toward such a bias toward intermediate connectivity, why would it evolve? Evolution can only think short term. I'll give you that an encoding bias might override the fitness gradient by making it impossible to follow, but I don't buy that evolution can magically figure out biases that are helpful in the long-term if the short-term fitness gradients all point in another direction. Or, at least, that's something we hope that evolution might do, but it's a controversial, unproven, and theoretically tricky issue that I certain don't think we can bank on happening until we understand it a lot more.
          > >
          > > > While you worry that modularity might "evolve away," the idea that it cannot evolve away to varying degrees sounds worse to me. Natural evolution is generally good about not keeping all its eggs in one basket - a trait may evolve away in some branches but not in others. But for you to make an all-out attempt to bar such a deviation from square one is making a lot of strong assumptions about what we want to see 1 billion years in the future.
          > > >
          > > >
          > > I'd argue that your bias in the first replicator without any sustained fitness pressure would have zero effect on creatures a billion years later, especially in the case where there is an active fitness gradient by default away from modularity (which we know there is, since modularity never evolves without a bias or fitness pressure). Evolution would not keep any eggs in a basket with an active fitness penalty, at least not for a billion years. My 1-billion-years-later influence may thus be imperfect, but it at least exists!
          > >
          > > > So I'm still a fan of manipulating encoding over manipulating fitness. But I would not entirely despair on fitness because there will still be cases where there is no clear option for manipulating the encoding. But such scenarios are not ones we should be hoping for.
          > > >
          > > >
          > > I do think you make some great arguments for encodings, but I cannot envision a case in which an initial bias only makes a huge difference in the long-term. That's true even if the bias is neutral with respect to fitness (because it would drift away), but seems a certainty to me if it relates to a bias that has an active fitness penalty. The whole point of canalization is that it figures out what produces fit offspring and generates that type of organism: if modular creatures are less fit in the short term, evolution will canalize away from modularity, not toward it. That said, as I mentioned at the beginning of this conversation, I do think that a sustained encoding bias of some sort is an interesting approach that could work, although it may be that all we have to do is provide a fitness cost and then the encoding will canalize in a way that produces such a sustained bias. :-)
          > >
          > > As always, an interesting conversation!
          > > Best,
          > > Jeff
          > >
          > >
          > > > Best,
          > > >
          > > > ken
          > > >
          > > > --- In neat@yahoogroups.com, Jeff Clune wrote:
          > > > >
          > > > > Hello all,
          > > > >
          > > > > As Ken mentioned, we've discussed these issues in private. I'm going to include some of my comments from one of those email threads with slight modification, as I believe they summarize the views of Jean-Baptiste and I on the issues Ken raises. I'll then respond to a few individual comments by Ken afterwards.
          > > > >
          > > > > ---------------
          > > > > Ken,
          > > > >
          > > > > It's great to hear your feedback on our paper. Thanks for sending it.
          > > > >
          > > > > First off, thanks for the kind words. We're very glad you liked the paper and think it is important.
          > > > >
          > > > > Regarding a selection pressure vs. an encoding bias. We're not convinced that an initial encoding bias is a good way to encourage properties that one wants in phenotypes throughout evolution, such as modularity. If there is any deceptiveness (or even neutrality) regarding modularity at any point during the run then the bias will disappear, and then for the rest of evolutionary time nothing will encourage modularity. We are more convinced of the power of mutational bias in the encoding (i.e. a constant encoding bias instead of just an initial encoding bias), and we think it would be interesting to investigate area. However, if the encoding bias is under selection, then you have the same issue where it might evolve away. Selective pressures are interesting because they are constant, so you're more likely to get what you want. That raises the point you mention about our pressure being too strong, such that evolution could not deviate when it would be beneficial not to have modularity. That might be a problem if the pressure is too strong, but it seems likely that in many cases the benefits in terms of performance for being non-modular will outweigh the cost. In other words, evolution can decide to pay the cost of non-modularity when it is useful (e.g. in your example of a hub of connections between modules).
          > > > >
          > > > > Regarding playing with an encoding being safer than playing with selection pressures. Our view is that both are very complicated and can have unintended consequences, so playing with one is just as bad as the other. I think our field is more familiar with unintended consequences of selective pressures just because we historically tend to play with them more (and make simple encodings), but it is also very hard to intuit the consequences of choices regarding biases in complex encodings. In your case the consequences are relatively intuitive, precisely because they are so minimally interventionist...but that is also why I think they are not strong enough to cause modularity except in cases (like retina) where all you have to do is initially place evolution in the right attractor basin.
          > > > >
          > > > > Regarding the resource hog waste of having a cost objective. I have to have a little fun here and point out the irony of the co-champion of novelty search worrying about the resources consumed by non-high-performing individuals! Hehe. As you've persuaded me, I'm more interested in an algorithm that is interesting or that works than spending a little computation inefficiently.
          > > > >
          > > > > I'd also like to point out an innovation we came up with to mitigate the problem of preventing evolution from exploring solutions that are contrary to one of the objectives. We recognized that the cost objective is ultimately less important than the performance objective. We wanted evolution to periodically ignore the cost objective to explore stepping stones that had higher connectivity. To do that, we invented a technique that involves "probabilistic pareto dominance", wherein secondary objectives (in this case cost) are factored into pareto dominance only a small percentage of the time. That won't solve the problem you mention if you have to take a long, many-multi-generational walk through high-connectivity areas of the search space, but it does allow quick forays into that terrain without any fitness penalty. This technique could be used for any cost (or other) objective, so it is not specific to connectivity costs.
          > > > >
          > > > > See below for a few specific responses to your comments. I should note that below this the thoughts are my own and Jean-Baptiste should not be blamed for any of them! (Feel free to blame him for things above this line…we went over that text together a while back). ;-)
          > > > >
          > > > > > More generally the issue is the usual problem of deception, which is compounded by anything you do with fitness. For example, in a complex search space, there is a reasonable chance that the stepping stone to a good low-connectivity solution is something with higher connectivity. By manipulating fitness, you are cutting out all chances of encountering such a deceptive stepping stone. But even if you don't believe that could be true, the single-mindedness of always favoring low-connectivity could deceive you from many parts of the search space that might be stepping stones to something worthwhile, relating to connection density or not.
          > > > > >
          > > > > >
          > > > > True. But the same exact thing can be said for biases in the encoding: they prevent you from searching large areas of the search space. You may reply that it is only a bias, not a strict ban, but of course we know that in large search spaces biases hugely affect the landscape such that certain areas will practically never be visited.
          > > > >
          > > > > > On the other hand, manipulating the encoding is different because in effect it actually reorganizes the structure of the search space itself, which seems to me a more principled thing to do (if you can figure out a way to do it). Because the thing is, in that case, you do not need to worry about a permanent dead weight taking up some proportion of your population forever. Instead, while the encoding may *tend* to produce e.g. low-connectivity solutions, it can still escape that tendency without any penalty to fitness.
          > > > > >
          > > > > My instincts tell me that we create dead weight with encoding biases too. For example, an overly regular generative encoding (e.g. context free L-systems) is great if good solutions are perfectly regular, but if what is required is a mix of regularity and irregularity, then you spend your entire time producing only highly regular phenotypes that never wander into the appropriately irregular areas of the search space. Our IEEE TEC paper, for example, shows that HyperNEAT can spend thousands of generations spinning its wheels never generating solutions that HybrID could easily generate, demonstrating a "overly regular" dead weight associated with the biases of even the best known* generative encoding! Both fitness penalties and biases can cause you to focus your search in unproductive areas…which is ultimately what dead weight is.
          > > > >
          > > > > * in our opinion! :0)
          > > > >
          > > > > > Furthermore, in reality the best situation regarding modularity and connectivity is probably rather subtle, with most of the brain respecting the principle of low connectivity, but with a number of critical exceptions in key areas, such as major inter-module hubs. A sophisticated encoding can allow its bias to bend to make such nuanced exceptions (e.g. based on locations within a geometry), whereas a fitness penalty is a heavy hand and blunt instrument that cannot but help always to demand global and holistic subservience to dogmatic universals (unless you are willing to take a hit in fitness).
          > > > > >
          > > > > >
          > > > > I think the last clause you offer is the key exception though. As I mentioned above, if a certain phenotype pays for its wiring by increasing fitness, it can add high-connectivity areas anywhere that they are useful (without even needing to carve out that area in geometric space, which is often a difficult task for CPPNs). Instead of being a blunt instrument, a fitness penalty can be quite subtle, because it can allow connection-by-connection exceptions if they produce fitness improvements, and do so without any search overhead.
          > > > >
          > > > > > An interesting question in nature (where our brains evolved modular structure) is whether its tendency towards low connectivity is a result of an aspect of fitness in the wild, or an aspect of encoding bias. I think there is a lot of room in this question for arguing either way, but my hunch is that the bias is mostly in the encoding. My logic is that I think the reason that the connectivity of the brain is so much lower than what it could be (e.g. it is a tiny fraction of everything-to-everything connectivity) is an artifact of physics rather than an artifact of fitness. It is simply physically impossible for a giant 100-billion-to-100-billion connectivity to fit in a head anything close to our size. And physical impossibility is in some sense a property of encoding. That is, mutations that could step from a low-connectivity brain to a high one are few and far between simply because of physical constraint. So high-connectivity structures are simply a very small part of the search space of brains in the physical universe. However, at the same time, you can still get long-range connections from time to time because there is no universal penalty for doing so, just a lower a priori probability of such mutations occurring.
          > > > > >
          > > > > >
          > > > > Here I completely disagree with you. I see the force preventing the volume of neural connections from getting too large as a direct fitness cost, not an encoding bias. If mutations increase the size of the head, the baby and the mother are more likely to die in childbirth. Anthropologists have long known that evolution's desire to have larger and larger brains is the main reason why humans have such ridiculously high maternal and infant mortality "in the wild" (pre modern health care, and even post). Our encoding keeps producing such mutants, and it's death (via the physical constraints of the pelvis) that keep them from being kept around. The historical accident of birthing through the pelvis aside, there would still be fitness consequences for more vastly neural connections (e.g. neurons are metabolically expensive, neural connections require energy to build and maintain, and housing such large brains would create a large, clunky bobble head of a being that would be ungainly). It is possible that low connectivity is an encoding bias, but were that true I think it would look like some sort of growth rule that only grew a few connections per neuron (e.g. 10k). Evolution could have learned such a rule, and canalized that rule in a way that makes it unlikely to have mutations that produce orders of magnitude more neurons, but my guess is that if it has done so, it was because of the fitness costs associated with large numbers of neurons, not because of evolvability (or due to some historical accident). I do think it is interesting to study whether such biases exist in the encoding of neural growth rules, but even if they do exist I don't think that shows that a fitness cost was not the ultimate cause. Note: I recognize that you admit that this could be argued "either way"…are these the sorts of arguments you envisioned as being the other way?
          > > > >
          > > > > > In summary, the key difference between the alternatives is that with fitness you are saying "stay out of this part of the search space" whereas with encoding you are saying "this part of the search space is much smaller and hence less likely to encounter."
          > > > > >
          > > > > >
          > > > > I don't precisely understand your latter clause, but I think I understand the spirit of it. I disagree, though. The reasons biases work is because they do bias search towards some areas and away from others: so I think both encoding biases and fitness penalties have similar effects in this regard.
          > > > >
          > > > > > So, my speculation is that if you want to bias the search in highly complex domains, the best way is through the encoding. Fitness is a nasty quagmire that is deceptively tempting to manipulate, but never plays by the rules you wish it would. Of course, these are merely my own unproven intuitions and their veracity remains to be demonstrated. But at least it's something to think about.
          > > > > >
          > > > > >
          > > > >
          > > > > Thanks again for your feedback. As always, I appreciate your input and enjoy discussing these fascinating subjects.
          > > > >
          > > > > I want to end by clarifying that I agree that there are positives and negatives to both approaches. I do not see it as such an obvious choice between the two as you do. As I mentioned up top: I definitely don't think initial encoding biases that selection can get rid of will get us very far. For simple problems they will work, but for any challenging problem the initial bias will have long disappeared by the time it will matter. What we need is some constant force encouraging search to take promising paths. A fitness cost is one way to do that, but a *constant* (or, at least, *periodic*) encoding bias could do that just as well, and perhaps better.
          > > > >
          > > > >
          > > > > Best regards,
          > > > > Jeff Clune
          > > > >
          > > > > Assistant Professor
          > > > > Computer Science
          > > > > University of Wyoming
          > > > > jeffclune@
          > > > > jeffclune.com
          > > > >
          > > > >
          > > > > > Best,
          > > > > >
          > > > > > ken
          > > > > >
          > > > > > --- In neat@yahoogroups.com, Alexandre Devert wrote:
          > > > > > >
          > > > > > > Hi,
          > > > > > >
          > > > > > > Â Simple, clean experiment, with sharp results, congrats on that, definitely
          > > > > > > a step forward ! Of course, it begs for more questions. I would love to hear
          > > > > > > you on such (fairly open) questions
          > > > > > >
          > > > > > > Â Â 1) Do you think that selection pressure for low connectivity is sufficient in
          > > > > > > itself to evolve large coherent networks, or is it just a piece of the puzzle ?
          > > > > > > Â Â 2) Do you see your work as an indication that any approach biased to low
          > > > > > > connectivity would reproduce the result ? Or does the way you guys enforced
          > > > > > > this bias matters ?
          > > > > > >
          > > > > > > To meÂ
          > > > > > > 1) => Part of the puzzle. Should see how well it scales for increasingly
          > > > > > > complex task, when the connection graph gets bigger. A randomizedÂ
          > > > > > > search process on large graph sounds not so efficient, need something to guide it.
          > > > > > > I advocate construction process that have a feedback from what the neuronÂ
          > > > > > > network is computing. Don't know how to do it without creepling computational
          > > > > > > cost tho...
          > > > > > > 2) => I guess that the bias alone is enough, the way to introduce it might
          > > > > > > not be such a big deal.Â
          > > > > > >
          > > > > > > Again, great work, very helpful contribution :)
          > > > > > >
          > > > > > > Alex
          > > > > > > Â
          > > > > > > Dr. Devert Alexandre
          > > > > > > Researcher at the Nature Inspired Computation and Applications Laboratory (NICAL)
          > > > > > > Lecturer at School Of Software Engineering of USTC
          > > > > > > ----------------------------------------------------
          > > > > > > Homepage :Â http://www.marmakoide.org
          > > > > > > ----------------------------------------------------
          > > > > > > 166 Renai Road, Dushu Lake Higher Education Town
          > > > > > > Suzhou Industrial Park,
          > > > > > > Suzhou, Jiangsu, People's Republic of China
          > > > > > >
          > > > > > >
          > > > > > > ________________________________
          > > > > > > From: Jeff Clune
          > > > > > > To: neat users group group
          > > > > > > Cc: Jean-Baptiste Mouret ; Hod Lipson
          > > > > > > Sent: Thursday, February 7, 2013 1:57 AM
          > > > > > > Subject: [neat] New paper on why modules evolve, and how to evolve modular artificial neural networks
          > > > > > >
          > > > > > >
          > > > > > > Â
          > > > > > > Hello all,
          > > > > > >
          > > > > > > I'm extremely pleased to announce a new paper on a subject that many--including myself--think is critical to making significant progress in our field: the evolution of modularity.Â
          > > > > > >
          > > > > > > Jean-Baptiste Mouret, Hod Lipson and I have a new paper thatÂ
          > > > > > >
          > > > > > > 1) sheds light on why modularity may evolve in biological networks (e.g. neural, genetic, metabolic, protein-protein, etc.)
          > > > > > >
          > > > > > > 2) provides a simple technique for evolving neural networks that are modular and have increased evolvability, in that they adapt faster to new environments. The modules that formed solved subproblems in the domain.Â
          > > > > > > Cite: Clune J, Mouret J-B, Lipson H (2013) The evolutionary origins of modularity. Proceedings of the Royal Society B. 280: 20122863. http://dx.doi.org/10.1098/rspb.2012.2863%c3%82 (pdf)
          > > > > > >
          > > > > > > Abstract: A central biological question is how natural organisms are so evolvable (capable of quickly adapting to new environments). A key driver of evolvability is the widespread modularity of biological networksâ€"their organization as functional, sparsely connected subunitsâ€"but there is no consensus regarding why modularity itself evolved. Although most hypotheses assume indirect selection for evolvability, here we demonstrate that the ubiquitous, direct selection pressure to reduce the cost of connections between network nodes causes the emergence of modular networks. Computational evolution experiments with selection pressures to maximize network performance and minimize connection costs yield networks that are significantly more modular and more evolvable than control experiments that only select for performance. These results will catalyse research in numerous disciplines, such as neuroscience and genetics, and enhance our ability to harness
          > > > > > > evolution for engineering purposes.
          > > > > > >
          > > > > > > Video:Â http://www.youtube.com/watch?feature=player_embedded&v=SG4_aW8LMng
          > > > > > >
          > > > > > > There has been some nice coverage of this work in the popular press, in case you are interested:
          > > > > > > National Geographic:Â http://phenomena.nationalgeographic.com/2013/01/30/the-parts-of-life/MIT's Technology Review:Â http://www.technologyreview.com/view/428504/computer-scientists-reproduce-the-evolution-of-evolvability/%c3%82 Fast Company:Â http://www.fastcompany.com/3005313/evolved-brains-robots-creep-closer-animal-learningCornell Chronicle:Â http://www.news.cornell.edu/stories/Jan13/modNetwork.htmlScienceDaily:%c3%82 http://www.sciencedaily.com/releases/2013/01/130130082300.htm
          > > > > > >
          > > > > > > Please let me know what you think and if you have any questions. I hope this work will help our field move forward!
          > > > > > >
          > > > > > >
          > > > > > >
          > > > > > >
          > > > > > > Best regards,
          > > > > > > Jeff Clune
          > > > > > >
          > > > > > > Assistant Professor
          > > > > > > Computer Science
          > > > > > > University of Wyoming
          > > > > > > jclune@
          > > > > > > jeffclune.com
          > > > > > >
          > > > > >
          > > > > >
          > > > > >
          > > > >
          > > >
          > > >
          > >
          >
        • Jeff Clune
          Hello Martin, I just discovered this very nice paper that documents how modularity is ubiquitous in both vertebrate and animal brains/central nervous systems
          Message 4 of 30 , Feb 24, 2013
          • 0 Attachment
            Hello Martin,

            I just discovered this very nice paper that documents how modularity is ubiquitous in both vertebrate and animal brains/central nervous systems (and not just in the neocortex of vertebrates), and it describes (in more detail than I can understand) how such modules are identified:


            As such, it seems like the perfect paper to shed light on the questions you and I were discussing at the end of our conversation. 

            If you read it, can you let me/us know what you think about the issue of whether modules can be identified by neural wiring diagrams, via other aspects of cytoarchitecture, or both? I still also don't really understand what cytoarchitecture is and how it differs from wiring, so if you find any text that explains that for non-neuroscientists I would appreciate reading it. Is it just that the types of cells (or cellular parts) matter, not the wiring itself, such that you can see modules of cell type A as distinct from cell type B, but they are all wired up such that a wiring diagram would make them look non-modular when in reality there are distinct clusters if you look at some attribute of the cells?

            Best regards,
            Jeff Clune

            Assistant Professor
            Computer Science
            University of Wyoming



            On Thu, Feb 14, 2013 at 2:48 PM, martin_pyka <martin.pyka@...> wrote:
             

            > Another definition I use is a functionally encapsulated unit, with high intra-modular interactions and few inter-modular interactions. That is consistent with the Kashtan and Alon definition you mention, but it is not specific to networks. See the Lipson paper we cite for a longer discussion of these definitions.
            >
            > How would using the cytoarchitecture change our measure of modularity? My guess is that you'd end up with roughly the same identified modules, but that may be because I don't understand what you are suggesting.

            I might be wrong here but i think - on the physiological level - most parts of the neocortex can't be separated into modules (or areas) by looking for high intra-modul connectivity and low inter-modul connectivity. I thought, it's more the different cell types and their layering (e.g. Brodmann Areas) that allow us to define several anatomical areas within the cortex (which can also be related to different functions). But I am not an expert in anatomy.

            >
            > > 2. Emergence of modularity: independent from the definition, I don't see, why network modules should evolve in different ways like other organic structures. To my mind, the same principles that lead to the formation of global patterns in the organism contribute also to the formation of specific brain structures (favoring to my mind the cytoarchitectural perspective).
            > >
            > Did you get the impression from somewhere that I believe that network modules evolve differently than modules in other structures? I have not stated that (that I can recall). I think it is possible that this connection cost idea may also apply to many different types of structures (e.g. organs and kinematic linkages).

            ok, I think I misunderstood here something. Influenced by the paper of van Ooyen, I regarded the number of connections in a network as a property that arises from purely self-organizing principles (which are only little influenced by the encoding). From this perspective, your paper would state that modules can arise without any definition of modules in the genetic code (arguing against global pattern formation as the driving factor for the evolutionary origin of modules). However, I think you regard the number of connections (and possibly the targets of connections) as part of the genetic code (as you do it in your simulations). In this case, connection cost influences the genetic code towards lower connections and thereby producing an encoding for modules.

            This leads to the question, how much depends the number of connections in a neural network on the encoding (and how much just on environmental factors / physical laws)?


            >
            > > Clearly influenced by the cited articles, my humble and unproven opinion is that modularity emerges as a side-effect of the fact that sensors and motors already represent something like modules and that self-organizing mechanisms and mechanisms to evolve and alter global patterns contribute to the elaboration of these modules.
            > >
            > >
            > The brain is a huge collection of 100 trillion connections and billions of neurons…very few of those directly connect to external sensors and actuators. Do you think that spatial segregation of sensors and actuators drives the evolution of modularity within the brain itself (e.g. in the neo-cortical column, a module that is repeated over and over inside the cortex)?

            No, I don't think that :o). I was just refering to the title of your paper and hypothesizing that "the evolutionary origin of modularity" may come from the modularity of the rest of the organism.


            > I think it is possible that this connection cost idea may also apply to many different types of structures (e.g. organs and kinematic linkages).

            Yes, maybe. Have you followed the discussions in the article of your paper over at technologyreview.com? There are some nice comments relating this work to the second law of thermodynamics (and suggesting you for the Nobel Price :)). Indeed, physical limitations in the number of connections (in any system) constrain the search space in evolution. Still, I don't see how a factor - like connection cost - could facilitate the emergence of large scale networks (e.g. something like different cytoarchitectures). Therefore, I am wondering what kind of other factors drive modularization.

            Anyway, I really enjoy thinking about these kinds of questions and discussing them here with you. This is normally something I am looking for when I am visiting conferences. It's good to have this forum here to debate these topics without time pressure.

            Martin


          • martin_pyka
            Hi Jeff, thanks for the article. I enjoyed reading it although the level of detail was also for me partly too much. ;) Before I comment on it and other stuff
            Message 5 of 30 , Feb 26, 2013
            • 0 Attachment
              Hi Jeff,

              thanks for the article. I enjoyed reading it although the level of detail was also for me partly too much. ;)

              Before I comment on it and other stuff let me just recall and separate the questions that we discussed.

              A) What is the ==> origin <== of modularity in networks (e.g. CNS)?

              B) What criteria are helpful to identify modules in general?

              Let me break down question B to
              B1) On which level do we want to identify modules?
              B2) Should a module represent a functional unit?

              In my last posting I claimed that using the degree of connectivity to identify modules in natural neural networks might *not always* reveal modules that have been identified due to functional or anatomical criterias. But it depends, in my opinion, on your focus and on which level of detail you are looking at

              Let's say: B1: Low level of detail, B2: Yes.
              I suppose that the columnar organisation in sensory and motor areas (as described in the review by Leise) can be very well detected by your connectivity criteria. For associative areas, it is more complicated, I guess. It is not clear, whether a cortical column represents really a functional unit (see for example http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1569491/) and it turned out to be difficult to identify "cell assemblies" that behave according to a "representational framework" (A good introduction is the first page of this review: http://www.ncbi.nlm.nih.gov/pubmed/21040841).

              Here is an example of a subcortical region, where the connectivity criteria would fail. Three important regions of the hippocampus are called Dental Gyrus, CA3 and CA1. Neurons in CA 1 receive inputs from many regions and project to some regions but they don't have recurrent connections (http://en.wikipedia.org/wiki/File:CajalHippocampus_(modified).png). CA 1 is defined according to anatomical properties (cell type, projections, location etc.) and seem to have a specific function. It has zero within-module connectivity and high inter-module connectivity. Therefore, it would be not identified as a module acccording to the connectivity criteria as each CA 1 neuron would be assigned either to CA3, DG, Entorhinal Cortex or Septum.

              Let's say: B1: Higher level of detail.

              In this case, I see it like you described it:

              > Is it just that the types of cells (or cellular parts) matter, not the wiring itself, such that you can see modules of cell type A as distinct from cell type B, but they are all wired up such that a wiring diagram would make them look non-modular when in reality there are distinct clusters if you look at some attribute of the cells?

              Cytoarchitecture refers to the cell types, their layering and wiring (e.g. http://en.wikipedia.org/wiki/Brodmann_area). I am simply not sure whether functions that have been assigned to large groups of columns could be detected with the connectivity criteria.

              If B2 is answered with No, all of this is doesn't matter of course.

              This leads to question B:
              I think the connectivity criteria is helpful and probably applicable on many local circuits, but the various ways functions (and thereby modules) are implemented call – in my opinion – for other profound design principles / constraints.

              Which leads to question A:
              In this regard, the review by Leise confirmed my opinion that modularity in a CNS arises from modularity of the rest of the organism. If CNSs developed really from skin brain organisms (see: http://www.nature.com/nrn/journal/v4/n8/abs/nrn1175.html) than the emergence of a CNS would in actual fact more represent a decrease in modularity.

              However, the fact that the number of connections increased more in proportion to the number of neurons (but not in proportion to the square of the number of neurons) might be something which can be reached in simulations by using connection cost in a fitness function or an encoding bias (and I personally thing it is a mixture of encoding and self-organizing mechanisms). But here I think, the field lacks a formal grounding to explain in an unambiguous way why someone should implement a bias in a particular manner.

              I am looking forward to your response :)

              Best,
              Martin
            • martin_pyka
              oh, I wish I could correct some things in my last posting. with low level of detail I simply mean low level or level of just a few neurons , while with
              Message 6 of 30 , Feb 26, 2013
              • 0 Attachment
                oh, I wish I could correct some things in my last posting. with "low level of detail" I simply mean "low level" or "level of just a few neurons", while with "high level of detail" I mean "high / abstract level", groups of thousands of neurons. I am sorry for the misleading phrasing.
              • Jeff Clune
                Hello Ken, Please see below. Note: I don t respond to everything you say if my reply would be something I ve already said (in this response or earlier). ... I
                Message 7 of 30 , Feb 26, 2013
                • 0 Attachment
                  Hello Ken,

                  Please see below. Note: I don't respond to everything you say if my reply would be something I've already said (in this response or earlier). 

                  Nature doesn't have anything analogous either, which means there is at least some evidence that the "fitness bias" analogy with nature is not lining up perfectly. You might point to the continuing existence of single-celled organisms as something similar to the perpetual dead-weight in this formulation, but they aren't really analogous because single-celled organisms are functional - they retain the ability to make copies of themselves and continue to evolve in their own right - while the low-connectivity deadweight maintains no capability whatsoever. On the other hand, suspiciously, as in nature, nothing similar to such a deadweight niche is perpetuated by a biased encoding.

                  That's also true, but that fault does not lie with the fitness cost concept, it lies with the fact that multi-objective algorithms, which are the cause of the dead weight, do not perfectly analogize to nature. They're just better than a weighted sum for other reasons, but the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue.  

                  Doesn't it seem a little strange that the price we have to pay to obtain modular structure is to maintain a perpetual dead pool of genetic junk? Note that it doesn't suggest that such a system won't work in some cases, but it's inelegant enough to raise questions about the best realization of the concept..


                  All I think that calls into question is the optimality of multi-objective algorithms when you don't want the extreme of one objective. But that problem almost always occurs in multi-objective algorithms, so your really indicting the whole field of MOEA instead of our approach of using a fitness penalty instead of a biased encoding, no?

                  No I don't think I'm indicting MOEAs in general.  The problem here is not some inherent problem with MOEAs, but that MOEAs were not designed for the purpose for which you have borrowed them.   MOEAs are designed to return a Pareto front given a number of objective trade-offs.  They are doing that perfectly well in your experiment.  It's just that your interest is not perfectly aligned with that design:  Your interest is in the "best tradeoff" (which is inherently slippery to formalize and not formalized by a traditional  MOEA), while some trade-offs that MOEAs are by design made to preserve and elevate (such as total dominance on low connectivity) are of almost no interest in this domain, as you've agreed.  So my general critique of manipulating the fitness function has nothing to do with MOEAs specifically.


                  I still think you are indicting all (or at least most) of MOEAs. In many applications (e.g. fitness and diversity, or solving task A and task B), we're rarely interested in the very extreme solutions that totally dominate on only one of the objectives. If we were interested in those organisms, we'd just be rewarding performance on that task and not use MOEAs at all. I feel it is the same story with a connection cost objective. 

                  Rather, what we are observing is simply the struggle to find some kind of expression of fitness that aligns with what you actually want to see (and thereby incentivizes following the right path through the search space).  And the fact that you settled on MOEAs as the best option, and that they do not align all that well, just illustrates how nasty this little problem of tweaking fitness really is. 

                  You noted that MOEAs are "just better than a weighted sum for other reasons," but that's exactly the problem.  Those "other reasons" are indeed unrelated to the problem you are trying to solve here.  That is, MOEAs are a blunt kludge in this particular context.  But although you defend fitness in general with the response that "the fitness cost concept could easily be implemented in a weighted sum fitness function and not have this dead weight issue," the problem there is that you once again end up with similarly unnatural and incongruous implications:

                  If you make fitness a weighted sum and one component of that weighted sum is "low connectivity," then there will *still* be a special eternally protected pocket for non-functional low-connectivity structure.  The reason is that if there is any progress on actually solving the problem (aside from connectivity), then those networks that are moving towards solving it will be more connected than the lowest possible connectivity.  Very often, because of mutation, some of these more connected networks will break, leading to an inevitable (and unavoidable) subpopulation  of higher-connectivity networks that are broken and nonfunctional.  Because of this inevitable subset of failures in every generation, a great strategy for some of the population is to stay as minimally connected as possible while ignoring functionality (just as in the MOEA) because they can be just barely  "good enough" to keep perpetuating merely by their connectivity fitness bonus versus those who are more connected and also nonfunctional.

                  Now you may then reason that there is some more complicated way to counteract this problem.  For example, you may say, well, we just need to be more strict about selection, so amp up selection pressure by blocking more of the population from reproducing.  But you have no way to know a priori what that threshold should be, so you are risking breaking evolution in other ways (such as becoming too convergent) by wiping out diversity.  In fact, early in evolution, when most structure is nonfunctional or less functional, that could short circuit the  whole process. 


                  As I have said repeatedly, I agree there are all sorts of nasty consequences and tricky issues when picking fitness functions: I just think those issues also exist with encodings (and that fitness penalties are at least ever-influencing instead of ephemeral, as  initial encoding biases are). 

                  But I want to emphasize that my point here has nothing to do with trying to figure out the right "trick" to get fitness to actually align properly with what we want to see, or with natural evolution of modularity.  If you notice, when you begin to talk about using probabilistic Pareto fronts or something like that, you are just playing again the same "let's tweak things around hoping to get it right" game.  And my point here is that that game is ultimately a game of futility.  Sure, in this very simply task (relative to finding a natural brain), virtually anything will work even if it is radically out of whack with nature, so you can convince yourself that this problem is possible to reconcile.  You can convince yourself that with enough tweaking you'll just write down the magic fitness/MOEA recipe that equals finding what you want.  But the search space is so incredibly complex that such a dream is virtually impossible.  Because if you think about it, as you make one tweak after another, as you fix one unintended consequence with yet another trick, what you are ultimately doing is describing the path through the search space itself.  In other words, the logical extension of such a process is simply to identify all the stepping stones from all possible random starting points to the objective and give higher fitness for each step in the chain, a task akin to applied omnipotence.  You might as well just build the solution by hand in that case, because you would know all the stepping stones to the solution anyway.  So while these fitness tricks may work for now, while we play in modest playgrounds, there are big warning signs looming in the future.


                  This all sounds like a justification for novelty search vs. target-based search, not an initial encoding bias vs. a bias via the fitness function. Everything you say seems to me to apply to biases in the encoding space too: you pick one bias to solve one problem, but that creates an unintended consequence, so you pick another (and maybe a temporal order, so that the first bias solves your first problem and then the second kicks in), etc. You're playing the same game, but just using different tools. 

                  We are in agreement that encodings matter.  Of course, that is part of why I'm saying they're important.  But what I'm really saying is that encodings are far better suited than fitness for the long haul, i.e. for doing something interesting over millions of generations, largely *because* a good encoding can canalize.  The trick behind radically successful long-term evolution, in my view, is to keep your options open.  You don't want to say X is clearly better than Y.  What you want to say is, if X and Y are fundamentally different, I want to check both X and Y and see where each of them lead and follow both branches if both are interesting.  But furthermore, if some property of X proves helpful in leading it to all kinds of cool stuff (or the same for Y), then let's start preserving (i.e. canalizing) that property THEN, when we observe its potential, rather than a priori from day one when we would have to be an omnipotent being to anticipate everything that might be useful.  And only encoding offers that possibility.  Fitness is all about making choices about priorities long before you have any idea what the options are.  Encoding with canalization is about giving evolution the choice to find the level that's right for it.  There is nothing analogous to canalization in fitness.


                  Fitness causes canalization, so it determines what type of canalization occurs. I think your argument does not fully embrace the important difference between short-term payoffs and long-term payoffs. If there is a short-term fitness penalty by default for a property (e.g. modularity), but we know that modularity is beneficial in the long run, you simply will not get canalization for modularity unless you change the default fitness penalty to a reward: in fact, you'll get the exact opposite (evolution will gain a bias against producing modular structures). Evolution does not have foresight. In this situation, how do you envision modularity to become canalized? The easiest way to get such canalization is to change the fitness equation so that properties that generate long-term evolvability are also rewarded in the short-term (not just the initial generations, but in the short-term throughout evolution). Note: I have some ideas for how evolution could canalize things that pay off in the long-term when they don't in the short-term, but they are just hypotheses at this point and remain unproven, and would only occur in rare circumstances.

                  Canalization is an intriguing issue that is not fully understood, but it does happen, clearly in nature, and even in CPPNs.  For example, one cool experiment I tried informally on Picbreeder was to keep regenerating new populations from the same starting image over and over again from both (1) a highly-evolved image and (2) a low-generation image.  Not surprisingly, the offspring of the highly-evolved image were significantly more consistent (based on my subjective perception) than of the less-evolved one.  That's canalization in action, even in Picbreeder.  By the way, applying an objective fitness function will undermine the potential for canalization (with CPPNs or anything else) because of the tendency of objectives to wreck the representation.  So fitness here has a huge disadvantage as a tool for manipulation:  Not only does it offer no mechanism remotely comparable to canalization, but it actually thwarts canalization in the encoding even if the encoding can do it.  It's like you keep whipping evolution for losing modularity and it keeps paying the price without every really getting the idea fundamentally (i.e. through the encoding).


                  There is always a fitness function, even in picbreeder and novelty search. A fitness function does not have to be a static target. I love all of your work on novelty search and its implications for how to revolutionize search and evolutionary algorithms, but there is no reason you can't hybridize a fitness penalty with those ideas (e.g. you could reward novelty but also have a connection cost). 

                  So this is why we want something like LEO where modularity *can* evolve away to different degrees:  We want to make no a priori assumptions about what must be correct in all generations, and instead allow evolution to try out everything.  And those things that lead to more potential over time will become canalized, you can be confident, and therein evolution works its magic. 


                  With a constant fitness penalty by default for modularity, the LEO bias towards short connections will disappear over time. Just simulate the experiment in your head: do you really think that if there is a constant, default, short-term fitness reward for entanglement/full-connectivity and you evolve networks for a billion years, HyperNEAT will hang on to that bias for short encodings? It may for early generations, but very quickly it will go away. 

                  So while you are saying regarding the human head size that DNA perhaps *can* encode brains with high connectivity but that fitness is preventing it, I would say that DNA *in general* perhaps can do that, but by now, long into the human lineage, these pseudo-modular structures are so deeply canalized that breaking out of that canal for the encoding would require herculean effort.  But that's beauty of encoding.

                  Agreed. But it learned that canalization not because it served a long-term evolvability benefit, but because of the short-term fitness cost of having more connections (e.g. the cost of a larger head). I think this is an example of a fitness penalty causing the canalization. 



                  That's why I don't like the word bias as much when it comes to encoding.  I'd think of it more as an option.  Evolution can try everything - preserve it in some lineages, to a less degree in others, and not at all in others still.  The ability to commit to subtlety is the magic here.  That's the smart strategy - keep your options open but also have the chance to commit to an option when it's working for you.  A single blunt bias imposed for a billion years (which is about closing off options) does not sound like a smart strategy in a search space more astronomical than imagination itself.


                  Options exist with fitness penalties too: you can pay the penalty in order to get a higher ultimate fitness. In the face of a constant short-term penalty, and without a counter-balancing fitness reward, evolution will quickly shut down options via canalization: so it will quickly learn for example to not produce modularity.  

                  I think your view of evolution here is too linear (as opposed to branching), which is why you tend to worry about things like what is good in the short term or long term.  But the idea that you can impose these blunt constraints on evolution for eternity fits with a linear perspective, so it makes sense if that's how you view it, as a kind of step-by-step succession along a single chain.  But I think evolution in nature is not walking this single tightrope towards some kind of near-optimal ideal, where all our effort needs to focus on not falling off the tightrope.   That is a very objective view, and I think the evidence is pointing more and more towards why that kind of perspective has been an impediment for us.  But I agree the conversation has been great and hope I made my points respectfully.   I realize we're wading into controversy and that reasonable minds can disagree here.


                  You assume that evolution will branch out and explore all these options even in the face of fitness penalties for that exploration. But that is not how evolution works (it may be how novelty search works, of course, which is why you have had some interesting results showing that evolution can take a more long-term view with respect to mutation rates). Evolution will tend to not explore paths that have lower fitness than the rest of the population, so you won't get the branching exploration you desire if there are default penalties for exploring certain areas.  

                  Maybe this is another way to think about it: you really dislike fitness penalties, but our work (and those before us) shows that there are all sorts of default fitness penalties, such as the default short-term fitness penalty against modularity. So, you already have a fitness penalty built in, and thus all of the problems you describe crop up: all our paper shows is that we should change the default fitness penalty to one more favorable to a property we care about for long-term evolvability, and voila….things work better. What do you think of that framing of the issue? 

                  I totally agree with you that we may be at a point where we understand each other's positions clearly, and still respectfully disagree. :-)


                  Best regards,
                  Jeff Clune

                  Assistant Professor
                  Computer Science
                  University of Wyoming
                  jeffclune@...
                  jeffclune.com



                Your message has been successfully submitted and would be delivered to recipients shortly.