Loading ...
Sorry, an error occurred while loading the content.

Re: [neat] Re: Design of Experiments Factory

Expand Messages
  • Colin Green
    Hi, ... Ok. I m still not 100% sure how specifically you re combining DoE with NEAT. Potentially I m misundertanding you at a fundamerntal level and the penny
    Message 1 of 9 , Jan 1, 2008
    • 0 Attachment
      Hi,

      On 25/12/2007, Ken Lloyd <kalloyd@...> wrote:
      > Ken and the group,
      >
      > I have a general idea for the DoE app that I'll go ahead and develop. I'll submit it
      > to the group, and if you find it useful - terrific.

      Ok. I'm still not 100% sure how specifically you're combining DoE with
      NEAT. Potentially I'm misundertanding you at a fundamerntal level and
      the penny will drop on seeing the code.

      Thanks for the positive feedback BTW, it's always nice to know people
      are using SharpNeat 'out there'.

      Colin.
    • afcarl2
      Ken Lloyd, Are you proposing that DOE be used to: a) pre-NEAT DOE to identify significant variables to be introduced to NEAT as inputs (a subset of possible
      Message 2 of 9 , Jan 2, 2008
      • 0 Attachment
        Ken Lloyd,

        Are you proposing that DOE be used to:

        a) pre-NEAT DOE to identify significant variables to be introduced to
        NEAT as inputs (a subset of possible NEAT inputs)?, or

        b) concurrent NEAT DOE to identify locally significant variables (a
        subset of current NEAT inputs) to an approximator of the fitness
        function?

        It seems that "a" is addressed to some degree with minimal
        connections of the starting genome. "b" is interesting from the
        perspective of applying some intelligence to fitness evaluation,
        since standard NEAT param values are grossly inefficient when it
        comes to value added by the default fitness evaluations. This becomes
        very evident when the computational requirements of the fitness
        evaluation become non-trival. Unfortunately, most practical problems
        that NEAT would be useful for fall into this catagory.

        Or, perhaps your proposal is different than both "a" or "b" above?
        Please advise.

        Thanks,
        Andy Carl


        --- In neat@yahoogroups.com, "Ken Lloyd" <kalloyd@...> wrote:
        >
        > Ken and the group,
        >
        > I have a general idea for the DoE app that I'll go ahead and
        develop. I'll
        > submit it to the group, and if you find it useful - terrific.
        >
        > It will be .NET in C#, and I'll keep it backwardly compatible to
        1.1.
        >
        > In the meantime - I'll take a look at the various configurators you
        > mentioned.
        >
        > Ken Lloyd
        >
        >
        > _____
        >
        > From: neat@yahoogroups.com [mailto:neat@yahoogroups.com] On Behalf
        Of
        > Kenneth Stanley
        > Sent: Monday, December 24, 2007 5:29 PM
        > To: neat@yahoogroups.com
        > Subject: [neat] Re: Design of Experiments Factory
        >
        >
        >
        > Ken, I think a GUI for setting up esperimental inputs and outputs
        > would be a nice contribution. One question is how much more you can
        > allow the user to cusomtize than just the inputs and outputs,
        > considering that the main part of an experiment is usually a type
        of
        > simulation used during evaluation. Given the breadth of possible
        > experiments, it's difficult to think of how to supply any general
        > sort of GUI for modifying experimental simulations. In any case,
        any
        > kind of GUI that adds an elemtn of easy customization would be cool.
        >
        > One thing you might want to take a look at for reference along
        these
        > lines is Charles Tarun's NERO Sensor Configuration Utility, a
        > standalone program made to let users customize NERO's robot sensors:
        >
        > http://z.cs.
        >
        <http://z.cs.utexas.edu/users/nn/nero/downloads/NeroSensorConfig.exe>
        > utexas.edu/users/nn/nero/downloads/NeroSensorConfig.exe
        >
        > Its documentation is here:
        >
        > http://z.cs.
        >
        <http://z.cs.utexas.edu/users/nn/nero/downloads/NEROSensorConfig.html>
        > utexas.edu/users/nn/nero/downloads/NEROSensorConfig.html
        >
        > (it also requires NET 1.1 Framework)
        >
        > By the way, another idea along similar lines that I think would be
        > very cool is a HyperNEAT substrate configurator, that lets you
        > graphically construct substrate geometry through a nice GUI. At
        this
        > point, no such utility exists anywhere that I know of.
        >
        > ken
        >
        > --- In neat@yahoogroups. <mailto:neat%40yahoogroups.com> com, "Ken
        Lloyd"
        > <kalloyd@> wrote:
        > >
        > > Colin,
        > >
        > > First, let me say that I am really impressed with SharpNEAT, and
        I'm
        > > learning a lot through discovering ways of using it.
        > >
        > > What I was thinking of developing was a GUI based app that would
        > allow (say)
        > > manufacturers to input throughput functions for various
        fabrication
        > cells in
        > > a graphical network, then find (one or more) system-wide
        > optimizations (as
        > > opposed to localized optimizations) for various product routings.
        > >
        > > Currently, the way one goes about creating experiments for NEAT
        is
        > to "hard
        > > code" them. Yes, these experiments are made available to NEAT by
        > using the
        > > IExperiment, and ISimulation interfaces (dependent upon the
        > > IActivationFunction) as well as other experimentally determined
        > parameters.
        > > I was thinking of creating an easy way for non-programmer types
        to
        > create
        > > Design of Experiments that can be saved in the XML format, then
        run
        > on (a
        > > distributed cluster of) NEAT dll's for comparison of
        > various "lessons
        > > learned" patterns.
        > >
        > > Make sense?
        > >
        > > The University of Edinburgh has an interesting system for
        > developing VLSI
        > > IC's that uses 3 applications (CAESAR, Calphurnia, and RS/1) that
        > takes CAD
        > > drawings as inputs, creating a DoE tree and simulating them. The
        > simulation
        > > is analytical. I was thinking about ways of using NEAT in a
        > similar system
        > > for pattern discovery.
        > >
        > > Ken
        > >
        > >
        > >
        > >
        > > _____
        > >
        > > From: neat@yahoogroups. <mailto:neat%40yahoogroups.com> com
        > [mailto:neat@yahoogroups. <mailto:neat%40yahoogroups.com> com] On
        Behalf
        > Of Colin
        > > Green
        > > Sent: Saturday, December 22, 2007 11:33 AM
        > > To: neat@yahoogroups. <mailto:neat%40yahoogroups.com> com
        > > Subject: Re: [neat] Design of Experiments Factory
        > >
        > >
        > >
        > > On 22/12/2007, Ken Lloyd <kalloyd@wattsys. <mailto:kalloyd%
        > 40wattsys.com>
        > > com> wrote:
        > > > Before I possibly re-invent the wheel, I thought I'd ask if
        > anyone has
        > > already developed a Design of Experiments interface to NEAT that
        > normalizes
        > > the experiment functions into inputs, biases and outputs? If not,
        > is this
        > > something that the group would find useful?
        > >
        > > If you're meaning code interfaces then yeh, sharpneat has an
        > > IExperiment interface, see:
        > >
        > > http://sharpneat.
        > >
        > <http://sharpneat.
        >
        <http://sharpneat.svn.sourceforge.net/viewvc/sharpneat/trunk/src/Sharp
        >
        > svn.sourceforge.net/viewvc/sharpneat/trunk/src/Sharp
        > NeatLi
        > > b/Experiments/IExperiment.cs?revision=6&view=markup>
        > >
        >
        svn.sourceforge.net/viewvc/sharpneat/trunk/src/SharpNeatLib/Experiment
        > s/IExp
        > > eriment.cs?revision=6&view=markup
        > >
        > > With traditional NEAT there is not really any point in having more
        > > than one bias input so I chose to fix it to node 0, hence that is
        > > assumed to always be present and isn't defined by IExperiment.
        > >
        > > Colin.
        > >
        >
      • Ken Lloyd
        I didn t mean to confuse everyone. Here are my general thoughts: Design of Experiments has proven to be valuable in solving linear problems of substantial
        Message 3 of 9 , Jan 3, 2008
        • 0 Attachment
          I didn't mean to confuse everyone.  Here are my general thoughts:
           
          Design of Experiments has proven to be valuable in solving linear problems of substantial size and coupling.  When networked problems are identified as non-linear, neural networks provide valuable insight into the problem domain.  There has been recent work integrating the two techniques using a "committee" of neural network evolutions. I was thinking that NEAT could provide insight into non-linear problem sets where there was not a lot of a priori information, or gathering that information would be expensive.  Identify emergent features from the grid of networks.
           
          c.f Coleman and Block "Non-linear experimental design using regularized Bayesian neural networks" (which focuses on more traditional feedforward ANNs)
           
          This whole idea may be orthogonal to the objectives of the NEAT group, however. I don't want to upset the good work you guys are doing.  As I'm kinda up to my ears, it will take me a while to put this together, but if the group is interested, I'd be happy to open source this.
           
          Hope everyone had a Happy New Year.
           
          Ken Lloyd
           


          From: neat@yahoogroups.com [mailto:neat@yahoogroups.com] On Behalf Of afcarl2
          Sent: Wednesday, January 02, 2008 12:06 PM
          To: neat@yahoogroups.com
          Subject: [neat] Re: Design of Experiments Factory

          Ken Lloyd,

          Are you proposing that DOE be used to:

          a) pre-NEAT DOE to identify significant variables to be introduced to
          NEAT as inputs (a subset of possible NEAT inputs)?, or

          b) concurrent NEAT DOE to identify locally significant variables (a
          subset of current NEAT inputs) to an approximator of the fitness
          function?

          It seems that "a" is addressed to some degree with minimal
          connections of the starting genome. "b" is interesting from the
          perspective of applying some intelligence to fitness evaluation,
          since standard NEAT param values are grossly inefficient when it
          comes to value added by the default fitness evaluations. This becomes
          very evident when the computational requirements of the fitness
          evaluation become non-trival. Unfortunately, most practical problems
          that NEAT would be useful for fall into this catagory.

          Or, perhaps your proposal is different than both "a" or "b" above?
          Please advise.

          Thanks,
          Andy Carl

          --- In neat@yahoogroups. com, "Ken Lloyd" <kalloyd@... > wrote:
          >
          > Ken and the group,
          >
          > I have a general idea for the DoE app that I'll go ahead and
          develop. I'll
          > submit it to the group, and if you find it useful - terrific.
          >
          > It will be .NET in C#, and I'll keep it backwardly compatible to
          1.1.
          >
          > In the meantime - I'll take a look at the various configurators you
          > mentioned.
          >
          > Ken Lloyd
          >
          >
          > _____
          >
          > From: neat@yahoogroups. com [mailto:neat@yahoogroups. com] On Behalf
          Of
          > Kenneth Stanley
          > Sent: Monday, December 24, 2007 5:29 PM
          > To: neat@yahoogroups. com
          > Subject: [neat] Re: Design of Experiments Factory
          >
          >
          >
          > Ken, I think a GUI for setting up esperimental inputs and outputs
          > would be a nice contribution. One question is how much more you can
          > allow the user to cusomtize than just the inputs and outputs,
          > considering that the main part of an experiment is usually a type
          of
          > simulation used during evaluation. Given the breadth of possible
          > experiments, it's difficult to think of how to supply any general
          > sort of GUI for modifying experimental simulations. In any case,
          any
          > kind of GUI that adds an elemtn of easy customization would be cool.
          >
          > One thing you might want to take a look at for reference along
          these
          > lines is Charles Tarun's NERO Sensor Configuration Utility, a
          > standalone program made to let users customize NERO's robot sensors:
          >
          > http://z.cs.
          >
          <http://z.cs. utexas.edu/ users/nn/ nero/downloads/ NeroSensorConfig .exe>
          > utexas.edu/users/ nn/nero/download s/NeroSensorConf ig.exe
          >
          > Its documentation is here:
          >
          > http://z.cs.
          >
          <http://z.cs. utexas.edu/ users/nn/ nero/downloads/ NEROSensorConfig .html>
          > utexas.edu/users/ nn/nero/download s/NEROSensorConf ig.html
          >
          > (it also requires NET 1.1 Framework)
          >
          > By the way, another idea along similar lines that I think would be
          > very cool is a HyperNEAT substrate configurator, that lets you
          > graphically construct substrate geometry through a nice GUI. At
          this
          > point, no such utility exists anywhere that I know of.
          >
          > ken
          >
          > --- In neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com, "Ken
          Lloyd"
          > <kalloyd@> wrote:
          > >
          > > Colin,
          > >
          > > First, let me say that I am really impressed with SharpNEAT, and
          I'm
          > > learning a lot through discovering ways of using it.
          > >
          > > What I was thinking of developing was a GUI based app that would
          > allow (say)
          > > manufacturers to input throughput functions for various
          fabrication
          > cells in
          > > a graphical network, then find (one or more) system-wide
          > optimizations (as
          > > opposed to localized optimizations) for various product routings.
          > >
          > > Currently, the way one goes about creating experiments for NEAT
          is
          > to "hard
          > > code" them. Yes, these experiments are made available to NEAT by
          > using the
          > > IExperiment, and ISimulation interfaces (dependent upon the
          > > IActivationFunction ) as well as other experimentally determined
          > parameters.
          > > I was thinking of creating an easy way for non-programmer types
          to
          > create
          > > Design of Experiments that can be saved in the XML format, then
          run
          > on (a
          > > distributed cluster of) NEAT dll's for comparison of
          > various "lessons
          > > learned" patterns.
          > >
          > > Make sense?
          > >
          > > The University of Edinburgh has an interesting system for
          > developing VLSI
          > > IC's that uses 3 applications (CAESAR, Calphurnia, and RS/1) that
          > takes CAD
          > > drawings as inputs, creating a DoE tree and simulating them. The
          > simulation
          > > is analytical. I was thinking about ways of using NEAT in a
          > similar system
          > > for pattern discovery.
          > >
          > > Ken
          > >
          > >
          > >
          > >
          > > _____
          > >
          > > From: neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com
          > [mailto:neat@ yahoogroups. <mailto:neat% 40yahoogroups. com> com] On
          Behalf
          > Of Colin
          > > Green
          > > Sent: Saturday, December 22, 2007 11:33 AM
          > > To: neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com
          > > Subject: Re: [neat] Design of Experiments Factory
          > >
          > >
          > >
          > > On 22/12/2007, Ken Lloyd <kalloyd@wattsys. <mailto:kalloyd%
          > 40wattsys.com>
          > > com> wrote:
          > > > Before I possibly re-invent the wheel, I thought I'd ask if
          > anyone has
          > > already developed a Design of Experiments interface to NEAT that
          > normalizes
          > > the experiment functions into inputs, biases and outputs? If not,
          > is this
          > > something that the group would find useful?
          > >
          > > If you're meaning code interfaces then yeh, sharpneat has an
          > > IExperiment interface, see:
          > >
          > > http://sharpneat.
          > >
          > <http://sharpneat.
          >
          <http://sharpneat. svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ Sharp
          >
          > svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ Sharp
          > NeatLi
          > > b/Experiments/ IExperiment. cs?revision= 6&view=markup>
          > >
          >
          svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ SharpNeatLib/ Experiment
          > s/IExp
          > > eriment.cs?revision =6&view=markup
          > >
          > > With traditional NEAT there is not really any point in having more
          > > than one bias input so I chose to fix it to node 0, hence that is
          > > assumed to always be present and isn't defined by IExperiment.
          > >
          > > Colin.
          > >
          >

        • Ken Lloyd
          This is a resend of my original reply. The original didn t seem to make it through. I didn t mean to confuse everyone. Here are my general thoughts: Design of
          Message 4 of 9 , Jan 4, 2008
          • 0 Attachment
            This is a resend of my original reply. The original didn't seem to make it through.
             
            I didn't mean to confuse everyone.  Here are my general thoughts:
             
            Design of Experiments has proven to be valuable in solving linear problems of substantial size and coupling.  When networked problems are identified as non-linear, neural networks provide valuable insight into the problem domain.  There has been recent work integrating the two techniques using a "committee" of neural network evolutions. I was thinking that NEAT could provide insight into non-linear problem sets where there was not a lot of a priori information, or gathering that information would be expensive.  Identify emergent features from the grid of networks.
             
            c.f. Gilardi and Faraj, "Design of Experiments by Committee of Neural Networks".
            c.f. Coleman and Block, "Non-linear experimental design using regularized Bayesian neural networks" (which focuses on more traditional feedforward ANNs)
             
            This whole idea may be orthogonal to the objectives of the NEAT group, however. I don't want to upset the good work you guys are doing.  As I'm kinda up to my ears, it will take me a while to put this together, but if the group is interested, I'd be happy to open source this.
             
            Hope everyone had a Happy New Year.
             
            Ken Lloyd
             
             
            =============================
            Kenneth A. Lloyd
            CEO and Director of Systems Science
            Watt Systems Technologies Inc.
            Albuquerque, NM USA


            From: neat@yahoogroups.com [mailto:neat@yahoogroups.com] On Behalf Of afcarl2
            Sent: Wednesday, January 02, 2008 12:06 PM
            To: neat@yahoogroups.com
            Subject: [neat] Re: Design of Experiments Factory

            Ken Lloyd,

            Are you proposing that DOE be used to:

            a) pre-NEAT DOE to identify significant variables to be introduced to
            NEAT as inputs (a subset of possible NEAT inputs)?, or

            b) concurrent NEAT DOE to identify locally significant variables (a
            subset of current NEAT inputs) to an approximator of the fitness
            function?

            It seems that "a" is addressed to some degree with minimal
            connections of the starting genome. "b" is interesting from the
            perspective of applying some intelligence to fitness evaluation,
            since standard NEAT param values are grossly inefficient when it
            comes to value added by the default fitness evaluations. This becomes
            very evident when the computational requirements of the fitness
            evaluation become non-trival. Unfortunately, most practical problems
            that NEAT would be useful for fall into this catagory.

            Or, perhaps your proposal is different than both "a" or "b" above?
            Please advise.

            Thanks,
            Andy Carl

            --- In neat@yahoogroups. com, "Ken Lloyd" <kalloyd@... > wrote:
            >
            > Ken and the group,
            >
            > I have a general idea for the DoE app that I'll go ahead and
            develop. I'll
            > submit it to the group, and if you find it useful - terrific.
            >
            > It will be .NET in C#, and I'll keep it backwardly compatible to
            1.1.
            >
            > In the meantime - I'll take a look at the various configurators you
            > mentioned.
            >
            > Ken Lloyd
            >
            >
            > _____
            >
            > From: neat@yahoogroups. com [mailto:neat@yahoogroups. com] On Behalf
            Of
            > Kenneth Stanley
            > Sent: Monday, December 24, 2007 5:29 PM
            > To: neat@yahoogroups. com
            > Subject: [neat] Re: Design of Experiments Factory
            >
            >
            >
            > Ken, I think a GUI for setting up esperimental inputs and outputs
            > would be a nice contribution. One question is how much more you can
            > allow the user to cusomtize than just the inputs and outputs,
            > considering that the main part of an experiment is usually a type
            of
            > simulation used during evaluation. Given the breadth of possible
            > experiments, it's difficult to think of how to supply any general
            > sort of GUI for modifying experimental simulations. In any case,
            any
            > kind of GUI that adds an elemtn of easy customization would be cool.
            >
            > One thing you might want to take a look at for reference along
            these
            > lines is Charles Tarun's NERO Sensor Configuration Utility, a
            > standalone program made to let users customize NERO's robot sensors:
            >
            > http://z.cs.
            >
            <http://z.cs. utexas.edu/ users/nn/ nero/downloads/ NeroSensorConfig .exe>
            > utexas.edu/users/ nn/nero/download s/NeroSensorConf ig.exe
            >
            > Its documentation is here:
            >
            > http://z.cs.
            >
            <http://z.cs. utexas.edu/ users/nn/ nero/downloads/ NEROSensorConfig .html>
            > utexas.edu/users/ nn/nero/download s/NEROSensorConf ig.html
            >
            > (it also requires NET 1.1 Framework)
            >
            > By the way, another idea along similar lines that I think would be
            > very cool is a HyperNEAT substrate configurator, that lets you
            > graphically construct substrate geometry through a nice GUI. At
            this
            > point, no such utility exists anywhere that I know of.
            >
            > ken
            >
            > --- In neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com, "Ken
            Lloyd"
            > <kalloyd@> wrote:
            > >
            > > Colin,
            > >
            > > First, let me say that I am really impressed with SharpNEAT, and
            I'm
            > > learning a lot through discovering ways of using it.
            > >
            > > What I was thinking of developing was a GUI based app that would
            > allow (say)
            > > manufacturers to input throughput functions for various
            fabrication
            > cells in
            > > a graphical network, then find (one or more) system-wide
            > optimizations (as
            > > opposed to localized optimizations) for various product routings.
            > >
            > > Currently, the way one goes about creating experiments for NEAT
            is
            > to "hard
            > > code" them. Yes, these experiments are made available to NEAT by
            > using the
            > > IExperiment, and ISimulation interfaces (dependent upon the
            > > IActivationFunction ) as well as other experimentally determined
            > parameters.
            > > I was thinking of creating an easy way for non-programmer types
            to
            > create
            > > Design of Experiments that can be saved in the XML format, then
            run
            > on (a
            > > distributed cluster of) NEAT dll's for comparison of
            > various "lessons
            > > learned" patterns.
            > >
            > > Make sense?
            > >
            > > The University of Edinburgh has an interesting system for
            > developing VLSI
            > > IC's that uses 3 applications (CAESAR, Calphurnia, and RS/1) that
            > takes CAD
            > > drawings as inputs, creating a DoE tree and simulating them. The
            > simulation
            > > is analytical. I was thinking about ways of using NEAT in a
            > similar system
            > > for pattern discovery.
            > >
            > > Ken
            > >
            > >
            > >
            > >
            > > _____
            > >
            > > From: neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com
            > [mailto:neat@ yahoogroups. <mailto:neat% 40yahoogroups. com> com] On
            Behalf
            > Of Colin
            > > Green
            > > Sent: Saturday, December 22, 2007 11:33 AM
            > > To: neat@yahoogroups. <mailto:neat% 40yahoogroups. com> com
            > > Subject: Re: [neat] Design of Experiments Factory
            > >
            > >
            > >
            > > On 22/12/2007, Ken Lloyd <kalloyd@wattsys. <mailto:kalloyd%
            > 40wattsys.com>
            > > com> wrote:
            > > > Before I possibly re-invent the wheel, I thought I'd ask if
            > anyone has
            > > already developed a Design of Experiments interface to NEAT that
            > normalizes
            > > the experiment functions into inputs, biases and outputs? If not,
            > is this
            > > something that the group would find useful?
            > >
            > > If you're meaning code interfaces then yeh, sharpneat has an
            > > IExperiment interface, see:
            > >
            > > http://sharpneat.
            > >
            > <http://sharpneat.
            >
            <http://sharpneat. svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ Sharp
            >
            > svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ Sharp
            > NeatLi
            > > b/Experiments/ IExperiment. cs?revision= 6&view=markup>
            > >
            >
            svn.sourceforge. net/viewvc/ sharpneat/ trunk/src/ SharpNeatLib/ Experiment
            > s/IExp
            > > eriment.cs?revision =6&view=markup
            > >
            > > With traditional NEAT there is not really any point in having more
            > > than one bias input so I chose to fix it to node 0, hence that is
            > > assumed to always be present and isn't defined by IExperiment.
            > >
            > > Colin.
            > >
            >

          Your message has been successfully submitted and would be delivered to recipients shortly.