Loading ...
Sorry, an error occurred while loading the content.

Re: [GP] Call for Hard Benchmark Problems in Genetic Programming

Expand Messages
  • david blane
    Good thread. I suggest y all start handicapping basketball.
    Message 1 of 11 , Mar 8 9:00 PM
    • 0 Attachment
      Good thread. I suggest y'all start handicapping basketball.

      On 8 March 2011 23:39, Lucas, Simon M <sml@...> wrote:


       just a small comment to add to this issue of
       speed:

       Sean, in your estimate you mentioned "single threaded" -
       but an inexpensive desktop machine now has quad cores
       and 8 parallel threads with hyper-threading - running a single threaded application
       only uses just over 10% of its power - soon running these
       apps multi-threaded will be the norm (maybe it already
       IS the norm for most people on this list...)

       cheers,
         Simon

      ps. I also agree this has been a good discussion.




      -----Original Message-----
      From: genetic_programming@yahoogroups.com [mailto:genetic_programming@yahoogroups.com] On Behalf Of Natalio Krasnogor
      Sent: 08 March 2011 17:05
      To: genetic_programming@yahoogroups.com
      Subject: Re: [GP] Call for Hard Benchmark Problems in Genetic Programming

      ....
      >
      > But I think speed is more important than you may have considered,
      > because there is an additional constraint:
      > benchmark comparisons require 50 to 100 independent runs per
      > treatment.
      >

      No, i did not ignore the above at all (I've been playing this game for a while now). Still, the constraints of 100 independent runs (how many runs you *really* need depends on -amogst others- the variance you get and the design of experiments you plan) or 0.004 seconds are pretty arbitrary numbers.
      At the end of the day the point is not to necessarily to compare GP versus GP' but to compare GP versus any other method, so if the same results on an expensive problem can be done with other methods I will take it any day.

      There are really expensive problems that may help push the GP state to the art considerably and I think these cannot be ignored. Often, what you get are situations not unlike design optimisation in which you are trying to get an optimal design for a very "unique" artifact that you will design once (and perhaps use multiple times or for long time), in that case, it does not really matter if the evaluation takes 2 years-cpu time (not a great deal nowadays anyways) because it gets recouped over time with the use of the designoid artifact. This situation is very different, of course, to one where for each, eg. new dataset, you need to GP-design a slightly different classifier of some (some examples have already been circulated in response to your original email covering this) in which of course you need speed.

      In any case we can talk more during gecco and it is gret that you have brought these matters of benchmarking to light, at the end of the day what would be fantastic is a well annotated taxonomy of benchmarks for various classes of scenarios people could draw from.

      Thanks.

      nat



      >
      >

      --


      Nat
      --

      -------------------------------------------------------------------------------------------
      NATALIO KRASNOGOR, Ph.D.

      Professor of Applied Interdisciplinary Computing Interdisciplinary Optimisation Laboratory (IOL) Automated Scheduling, Planning and Optimisation (ASAP) Group
      School of Computer Science
      Jubilee Campus, University of Nottingham Nottingham, NG81BB United Kingdom

      Tel.: +44 - (0)115 - 8467592
      Fax.: +44 - (0)115 - 8467066

      Skype: Natalio.Krasnogor

      URL: http://www.cs.nott.ac.uk/~nxk/

      e-mail: Natalio.Krasnogor@...

      ------------------------------------------------------------------------------------------

      Please consider sending your best papers to the Genetic and Evolutionary Computation Conference (GECCO) July 12-16 Dublin, Ireland, 2011 http://www.sigevo.org/gecco-2011/index.html






      ------------------------------------

      Yahoo! Groups Links





      ------------------------------------

      Yahoo! Groups Links

      <*> To visit your group on the web, go to:
         http://groups.yahoo.com/group/genetic_programming/

      <*> Your email settings:
         Individual Email | Traditional

      <*> To change settings online go to:
         http://groups.yahoo.com/group/genetic_programming/join
         (Yahoo! ID required)

      <*> To change settings via email:
         genetic_programming-digest@yahoogroups.com
         genetic_programming-fullfeatured@yahoogroups.com

      <*> To unsubscribe from this group, send an email to:
         genetic_programming-unsubscribe@yahoogroups.com

      <*> Your use of Yahoo! Groups is subject to:
         http://docs.yahoo.com/info/terms/



    • w langdon
      More toys: Order, Majority Computational Complexity Analysis of Simple Genetic Programming On Two Problems Modeling Isolated Program Semantics, FOGA 2011
      Message 2 of 11 , Mar 9 1:10 AM
      • 0 Attachment
        More toys: Order, Majority

        Computational Complexity Analysis of Simple Genetic Programming On Two
        Problems Modeling Isolated Program Semantics, FOGA 2011
        http://www.cs.bham.ac.uk/~wbl/biblio/gp-html/Durrett_2011_foga.html

        Bill
      • Una-May O'Reilly
        I think there s a distinction between model problems that we use for theoretical research eg ORDER and Majority, Max vs what we re in search of here:
        Message 3 of 11 , Mar 9 4:39 PM
        • 0 Attachment
          I think there's a distinction between "model" problems that we use for
          theoretical research eg ORDER and Majority, Max vs what we're in search of
          here: "benchmarks" that let GP extensions, alternatives, novelty be
          demonstrated, tested and compared.


          (And, I think "model" is more appropriate than "toy" when one considers
          what their authors designed them for.)

          Una-May



          w langdon wrote:
          >
          >
          > More toys: Order, Majority
          >
          > Computational Complexity Analysis of Simple Genetic Programming On Two
          > Problems Modeling Isolated Program Semantics, FOGA 2011
          > http://www.cs.bham.ac.uk/~wbl/biblio/gp-html/Durrett_2011_foga.html
          >
          > Bill
          >
          >

          --
          Una-May O'Reilly, PhD.
          Principal Research Scientist, CSAIL, MIT
          http://people.csail.mit.edu/unamay/
          http://groups.csail.mit.edu/EVO-DesignOpt
        Your message has been successfully submitted and would be delivered to recipients shortly.