Loading ...
Sorry, an error occurred while loading the content.

Easy-NN optimisation

Expand Messages
  • P. Dickens
    Hi Paul, Any ideas which settings could be different from the defaults in my case ? Within Controls (ACTION CHANGE CONTROLS) Use:- Learning Rate use
    Message 1 of 10 , Dec 5, 2012
    • 0 Attachment
      Hi Paul,

      Any ideas which settings could be different from the defaults in my case ?

      Within "Controls" (ACTION > CHANGE CONTROLS)
      Use:-
      Learning Rate use default
      Momentum use default
      Accelerator use default

      Use:- "Decay & Optimise"

      Within "Validating"
      Start by using 5% then take it up to 10 % of your data with a second
      network.
      This can be set as high as 40% in circumstances where you have abundant
      'Facts' but you don't.
      Your 'good' data (Facts) are somewhat limited so when you set your
      sample for testing count the number of 'good' Facts included in your
      testing datathat have been auto selected. If there is not enough i.e.
      5% or even 4% of available good Facts, then reset at say 8%, and
      then do a recount, if there are enough good Facts, then knock out enough
      of the bad Facts to take you back to a global 5%.

      In "Stops"
      Set all training errors to below 0.5 then aim to screw this down to 0.04
      as the net converges at each increment.
      Don't loose these Nets save a copy as a seperate name and test them.

      Set Stop when validating at 70% and gradually increase this when the
      Net achieves that result,
      & are within 10% then reduce this to say 5%.
      again don't loose these Nets save a copy as a seperate name and test them.

      Set your Stop if validating score is equal to:- say, if you have 10 set
      it at 7, then 8, then 9, then 10.
      Again this is a screwing down process, but don't loos the nets each time.

      One of these Nets may work a dream. Fully converged Nets are not always
      good. Sometimes the loosest looking Net works the best.

      If there is no good result from these Nets then:
      Set Up the network with one "Hidden layer".
      And start again as above.

      As an aside, something to consider, are there other inputs (columns) you
      could consider?

      P. Dickens.

      --------------------------------------------------------------------------------------
      Thank you Frank and aa99on,

      My app is most relevant to scenario 1.
      I have measurements of voltage, current, temperature and pressure.
      No greyhounds or pupils :-)
      The Input importance and sensitivity data indicates that all four inputs
      are useful.
      It could be that I need to have more learning cycles, in view of the
      72,000 rows.
      Any ideas which settings could be different from the defaults in my case ?
      Perhaps a specific number of cycles is not relevant here.
      Does anyone have any ideas on my original question ?

      Let me give another example in a medical application.
      A patient is monitored for temperature, blood pressure etc to decide
      whether they need a certain medication.
      The NN results have errors where :
      a) It reports medication needed when actually it is not. and . .
      b) It reports medication not needed when actually it is needed.

      As you can imagine, in most cases, (a) is much less serious than (b),
      and I believe this difference in the importance of the two error cases
      is quite common in real-world situations.

      Paul

      --- In easynn@yahoogroups.com <mailto:easynn%40yahoogroups.com>, "F"
      <lawism@...> wrote:
      >
      > Paul,
      >
      > Before suggesting anything, I first would prefer to know: Is your
      output-data such that it can be expressed in greater detail than only
      pass or fail? For example, students can pass/fail a test, yet they may
      also receive a score between 0-100.
      >
      > Frank
      >
      > --- In easynn@yahoogroups.com <mailto:easynn%40yahoogroups.com>,
      "lovellp" <lovellp@> wrote:
      > >
      > > Wonder if anyone can help ?
      > >
      > > I have a grid of approx 72,000 rows, 4 (real) input columns and one
      boolean output column.
      > >
      > > Approx 15% of the outputs are "pass" and the remainder "fail"
      > >
      > > After training, the validation obviously produces :
      > >
      > > a) Some "fail" outputs incorrectly reported as "pass." I shall call
      these FIRP, and . . .
      > > b) some "pass" outputs incorrectly reported as "fail" I shall call
      these PIRF
      > >
      > > Now, when I query with new data, I obviously want a small number of
      both FIRP and PIRF instances.
      > > However it is much more important (in my particular application) to
      reduce the number of FIRP instances, than the PIRF ones.
      > > In other words, the ratio FIRPs/PIRFs should be a minimum, but also
      the total of FIRPs + PIRFs should be as low as possible, for each 100
      validation examples.
      > >
      > > Now to my question.
      > > Is there any way I can change the training settings to minimize the
      number of FIRP cases (rather than both FIRP and PIRF)?
      > > Also, how does my FIRP and PIRF terminology relate to the data in
      the Diagnostic Array ?
      > >
      > > I am using Easy-NN ver 15.0a
      > > Any help would be much appreciated.
      > >
      > > Thanks,
      > > Paul


      [Non-text portions of this message have been removed]
    • Paul Lovell
      Thanks Frank, Easy-NN plus (which I am using), has a diagnostic array, which gives statistical data where the output is boolean. This is:  True Posives, True
      Message 2 of 10 , Dec 6, 2012
      • 0 Attachment
        Thanks Frank,

        Easy-NN plus (which I am using), has a diagnostic array, which gives statistical data where the output is boolean.
        This is:  True Posives, True Negatives, False Positives, False Negatives.
        Essential data for my particular application, but only available for boolean output.
        If I make the output a real value, I get a reasonable validating %, but I can't translate this into False Negatives and False Positives etc, which is important in my application.
        Anyway, I'll try some of your suggestions and report back if I get some useful results.

        Paul

        --- On Mon, 3/12/12, F <lawism@...> wrote:

        From: F <lawism@...>
        Subject: Re: Easy-NN optimisation
        To: easynn@yahoogroups.com
        Date: Monday, 3 December, 2012, 19:15
















         









        Thanks Paul,



        To be clear:



        I don't mean to suggest that the outputs be normalized to certain values. I instead mean to suggest that the outputs remain as pure as possible for as long as possible. For example:



        Supposing school grade results again. The students receive grades ranging from 55 to 98 where 70 is passing. The professor, instead of storing the actual, un-adjusted grade in her NN, only stores pass/fail. Fine. But...



        How much more difficult would it be for you or I or a NN to predict outputs given (last column is the output):



        1,1,2

        3,4,7

        5,6,11



        versus (1 is pass, 0 is fail):



        1,1,0

        3,4,1

        5,6,1



        The T/F P/F output has less meaning than the actual output itself. The inputs are 'analog,' not 'digital.'



        My long-winded point is that, in my opinion, a NN can do the math of a function better if given the actual results of the function rather than the results of a function of the function (if >=70, then true...etc.)



        That said, an addition output of Pass/Fail can still be utilized. This has the benefit of a sort of "second opinion" or 'checksum.'



        In such a scenario, you may get a mixed decision:



        Grade predicted to be greater than 70 but P/F output says F; you can then error on whichever side you deem as 'caution.'



        To be even clearer:



        Suppose the professor only gives 10 point tests. The output column would range from 0-10. There is no need to artificially convoluted the output to be between 0-100.



        REGARDING YOUR ORIGINAL QUESTION:



        * Unfortunately, you have many entries, but one technique my be to double the rows that are more important. For example, if passing is more critical than failing, then double each passing entry, while having only one entry for each failing row. In my opinion, this will bias the NN towards passing.



        * I've already mentioned my advise regarding the output data.



        * Finally, I've found that more NN are better than one NN. You can split one of the inputs along some logical lines and may find that you'll receive quicker, and more accurate results. One NN for some of the data. Another NN for a different category, etc.



        * RE: "Also, how does my FIRP and PIRF terminology relate to the data in the Diagnostic Array ?"



        Unfortunately, I only use JustNN. I'm not familiar with the Diagnostic Array.



        Apologize for the length and hope you can understand my ramblings; let me know if you need more clarity as to what I'm trying to say.



        Frank



        --- In easynn@yahoogroups.com, "lovellp" <lovellp@...> wrote:

        >

        > Hi Frank,

        >

        > Yes, I suppose the output data could be represented on a scale of 0 to 100, with a "pass mark" of 50.

        > An ideas on how would this help ?

        >

        > Paul

        >

        > --- In easynn@yahoogroups.com, "F" <lawism@> wrote:

        > >

        > > Paul,

        > >

        > > Before suggesting anything, I first would prefer to know: Is your output-data such that it can be expressed in greater detail than only pass or fail? For example, students can pass/fail a test, yet they may also receive a score between 0-100.

        > >

        > > Frank

        > >

        > > --- In easynn@yahoogroups.com, "lovellp" <lovellp@> wrote:

        > > >

        > > > Wonder if anyone can help ?

        > > >

        > > > I have a grid of approx 72,000 rows, 4 (real) input columns and one boolean output column.

        > > >

        > > > Approx 15% of the outputs are "pass" and the remainder "fail"

        > > >

        > > > After training, the validation obviously produces :

        > > >

        > > > a) Some "fail" outputs incorrectly reported as "pass." I shall call these FIRP, and . . .

        > > > b) some "pass" outputs incorrectly reported as "fail" I shall call these PIRF

        > > >

        > > > Now, when I query with new data, I obviously want a small number of both FIRP and PIRF instances.

        > > > However it is much more important (in my particular application) to reduce the number of FIRP instances, than the PIRF ones.

        > > > In other words, the ratio FIRPs/PIRFs should be a minimum, but also the total of FIRPs + PIRFs should be as low as possible, for each 100 validation examples.

        > > >

        > > > Now to my question.

        > > > Is there any way I can change the training settings to minimize the number of FIRP cases (rather than both FIRP and PIRF)?

        > > > Also, how does my FIRP and PIRF terminology relate to the data in the Diagnostic Array ?

        > > >

        > > > I am using Easy-NN ver 15.0a

        > > > Any help would be much appreciated.

        > > >

        > > > Thanks,

        > > > Paul

        > > >

        > >

        >



























        [Non-text portions of this message have been removed]
      • Paul Lovell
        Many thanks , I ll experiment using your suggestions and report back with my findings. Paul ... From: P. Dickens Subject: Easy-NN
        Message 3 of 10 , Dec 6, 2012
        • 0 Attachment
          Many thanks ,

          I'll experiment using your suggestions and report back with my findings.

          Paul

          --- On Wed, 5/12/12, P. Dickens <pdickens@...> wrote:

          From: P. Dickens <pdickens@...>
          Subject: Easy-NN optimisation
          To: easynn@yahoogroups.com
          Date: Wednesday, 5 December, 2012, 18:42
















           









          Hi Paul,



          Any ideas which settings could be different from the defaults in my case ?



          Within "Controls" (ACTION > CHANGE CONTROLS)

          Use:-

          Learning Rate use default

          Momentum use default

          Accelerator use default



          Use:- "Decay & Optimise"



          Within "Validating"

          Start by using 5% then take it up to 10 % of your data with a second

          network.

          This can be set as high as 40% in circumstances where you have abundant

          'Facts' but you don't.

          Your 'good' data (Facts) are somewhat limited so when you set your

          sample for testing count the number of 'good' Facts included in your

          testing datathat have been auto selected. If there is not enough i.e.

          5% or even 4% of available good Facts, then reset at say 8%, and

          then do a recount, if there are enough good Facts, then knock out enough

          of the bad Facts to take you back to a global 5%.



          In "Stops"

          Set all training errors to below 0.5 then aim to screw this down to 0.04

          as the net converges at each increment.

          Don't loose these Nets save a copy as a seperate name and test them.



          Set Stop when validating at 70% and gradually increase this when the

          Net achieves that result,

          & are within 10% then reduce this to say 5%.

          again don't loose these Nets save a copy as a seperate name and test them.



          Set your Stop if validating score is equal to:- say, if you have 10 set

          it at 7, then 8, then 9, then 10.

          Again this is a screwing down process, but don't loos the nets each time.



          One of these Nets may work a dream. Fully converged Nets are not always

          good. Sometimes the loosest looking Net works the best.



          If there is no good result from these Nets then:

          Set Up the network with one "Hidden layer".

          And start again as above.



          As an aside, something to consider, are there other inputs (columns) you

          could consider?



          P. Dickens.



          ----------------------------------------------------------

          Thank you Frank and aa99on,



          My app is most relevant to scenario 1.

          I have measurements of voltage, current, temperature and pressure.

          No greyhounds or pupils :-)

          The Input importance and sensitivity data indicates that all four inputs

          are useful.

          It could be that I need to have more learning cycles, in view of the

          72,000 rows.

          Any ideas which settings could be different from the defaults in my case ?

          Perhaps a specific number of cycles is not relevant here.

          Does anyone have any ideas on my original question ?



          Let me give another example in a medical application.

          A patient is monitored for temperature, blood pressure etc to decide

          whether they need a certain medication.

          The NN results have errors where :

          a) It reports medication needed when actually it is not. and . .

          b) It reports medication not needed when actually it is needed.



          As you can imagine, in most cases, (a) is much less serious than (b),

          and I believe this difference in the importance of the two error cases

          is quite common in real-world situations.



          Paul



          --- In easynn@yahoogroups.com <mailto:easynn%40yahoogroups.com>, "F"

          <lawism@...> wrote:

          >

          > Paul,

          >

          > Before suggesting anything, I first would prefer to know: Is your

          output-data such that it can be expressed in greater detail than only

          pass or fail? For example, students can pass/fail a test, yet they may

          also receive a score between 0-100.

          >

          > Frank

          >

          > --- In easynn@yahoogroups.com <mailto:easynn%40yahoogroups.com>,

          "lovellp" <lovellp@> wrote:

          > >

          > > Wonder if anyone can help ?

          > >

          > > I have a grid of approx 72,000 rows, 4 (real) input columns and one

          boolean output column.

          > >

          > > Approx 15% of the outputs are "pass" and the remainder "fail"

          > >

          > > After training, the validation obviously produces :

          > >

          > > a) Some "fail" outputs incorrectly reported as "pass." I shall call

          these FIRP, and . . .

          > > b) some "pass" outputs incorrectly reported as "fail" I shall call

          these PIRF

          > >

          > > Now, when I query with new data, I obviously want a small number of

          both FIRP and PIRF instances.

          > > However it is much more important (in my particular application) to

          reduce the number of FIRP instances, than the PIRF ones.

          > > In other words, the ratio FIRPs/PIRFs should be a minimum, but also

          the total of FIRPs + PIRFs should be as low as possible, for each 100

          validation examples.

          > >

          > > Now to my question.

          > > Is there any way I can change the training settings to minimize the

          number of FIRP cases (rather than both FIRP and PIRF)?

          > > Also, how does my FIRP and PIRF terminology relate to the data in

          the Diagnostic Array ?

          > >

          > > I am using Easy-NN ver 15.0a

          > > Any help would be much appreciated.

          > >

          > > Thanks,

          > > Paul



          [Non-text portions of this message have been removed]



























          [Non-text portions of this message have been removed]
        • F
          Thanks for explaining Paul. There are a couple of work-arounds. One being complicated/laborious (using easyNN functions etc. on the grid). Here is the other:
          Message 4 of 10 , Dec 6, 2012
          • 0 Attachment
            Thanks for explaining Paul.

            There are a couple of 'work-arounds.' One being complicated/laborious (using easyNN functions etc. on the grid). Here is the other:

            Keep the boolean output. Simply add an additional output column. This column will contain the non-boolean, raw results. (This assumes that the diagnostic array can still 'work' on/with the boolean column.)

            For example:

            Input1,...,InputN,Raw grade (40-97) output,Pass/Fail output

            Additionally, I can explain how it may be best to run the NN on one output column at a time; if you like, and are going to try the above.

            Frank

            --- In easynn@yahoogroups.com, Paul Lovell <lovellp@...> wrote:
            >
            > Thanks Frank,
            >
            > Easy-NN plus (which I am using), has a diagnostic array, which gives statistical data where the output is boolean.
            > This is:  True Posives, True Negatives, False Positives, False Negatives.
            > Essential data for my particular application, but only available for boolean output.
            > If I make the output a real value, I get a reasonable validating %, but I can't translate this into False Negatives and False Positives etc, which is important in my application.
            > Anyway, I'll try some of your suggestions and report back if I get some useful results.
            >
            > Paul
            >
            > --- On Mon, 3/12/12, F <lawism@...> wrote:
            >
            > From: F <lawism@...>
            > Subject: Re: Easy-NN optimisation
            > To: easynn@yahoogroups.com
            > Date: Monday, 3 December, 2012, 19:15
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >  
            >
            >
            >
            >
            >
            >
            >
            >
            >
            > Thanks Paul,
            >
            >
            >
            > To be clear:
            >
            >
            >
            > I don't mean to suggest that the outputs be normalized to certain values. I instead mean to suggest that the outputs remain as pure as possible for as long as possible. For example:
            >
            >
            >
            > Supposing school grade results again. The students receive grades ranging from 55 to 98 where 70 is passing. The professor, instead of storing the actual, un-adjusted grade in her NN, only stores pass/fail. Fine. But...
            >
            >
            >
            > How much more difficult would it be for you or I or a NN to predict outputs given (last column is the output):
            >
            >
            >
            > 1,1,2
            >
            > 3,4,7
            >
            > 5,6,11
            >
            >
            >
            > versus (1 is pass, 0 is fail):
            >
            >
            >
            > 1,1,0
            >
            > 3,4,1
            >
            > 5,6,1
            >
            >
            >
            > The T/F P/F output has less meaning than the actual output itself. The inputs are 'analog,' not 'digital.'
            >
            >
            >
            > My long-winded point is that, in my opinion, a NN can do the math of a function better if given the actual results of the function rather than the results of a function of the function (if >=70, then true...etc.)
            >
            >
            >
            > That said, an addition output of Pass/Fail can still be utilized. This has the benefit of a sort of "second opinion" or 'checksum.'
            >
            >
            >
            > In such a scenario, you may get a mixed decision:
            >
            >
            >
            > Grade predicted to be greater than 70 but P/F output says F; you can then error on whichever side you deem as 'caution.'
            >
            >
            >
            > To be even clearer:
            >
            >
            >
            > Suppose the professor only gives 10 point tests. The output column would range from 0-10. There is no need to artificially convoluted the output to be between 0-100.
            >
            >
            >
            > REGARDING YOUR ORIGINAL QUESTION:
            >
            >
            >
            > * Unfortunately, you have many entries, but one technique my be to double the rows that are more important. For example, if passing is more critical than failing, then double each passing entry, while having only one entry for each failing row. In my opinion, this will bias the NN towards passing.
            >
            >
            >
            > * I've already mentioned my advise regarding the output data.
            >
            >
            >
            > * Finally, I've found that more NN are better than one NN. You can split one of the inputs along some logical lines and may find that you'll receive quicker, and more accurate results. One NN for some of the data. Another NN for a different category, etc.
            >
            >
            >
            > * RE: "Also, how does my FIRP and PIRF terminology relate to the data in the Diagnostic Array ?"
            >
            >
            >
            > Unfortunately, I only use JustNN. I'm not familiar with the Diagnostic Array.
            >
            >
            >
            > Apologize for the length and hope you can understand my ramblings; let me know if you need more clarity as to what I'm trying to say.
            >
            >
            >
            > Frank
            >
            >
            >
            > --- In easynn@yahoogroups.com, "lovellp" <lovellp@> wrote:
            >
            > >
            >
            > > Hi Frank,
            >
            > >
            >
            > > Yes, I suppose the output data could be represented on a scale of 0 to 100, with a "pass mark" of 50.
            >
            > > An ideas on how would this help ?
            >
            > >
            >
            > > Paul
            >
            > >
            >
            > > --- In easynn@yahoogroups.com, "F" <lawism@> wrote:
            >
            > > >
            >
            > > > Paul,
            >
            > > >
            >
            > > > Before suggesting anything, I first would prefer to know: Is your output-data such that it can be expressed in greater detail than only pass or fail? For example, students can pass/fail a test, yet they may also receive a score between 0-100.
            >
            > > >
            >
            > > > Frank
            >
            > > >
            >
            > > > --- In easynn@yahoogroups.com, "lovellp" <lovellp@> wrote:
            >
            > > > >
            >
            > > > > Wonder if anyone can help ?
            >
            > > > >
            >
            > > > > I have a grid of approx 72,000 rows, 4 (real) input columns and one boolean output column.
            >
            > > > >
            >
            > > > > Approx 15% of the outputs are "pass" and the remainder "fail"
            >
            > > > >
            >
            > > > > After training, the validation obviously produces :
            >
            > > > >
            >
            > > > > a) Some "fail" outputs incorrectly reported as "pass." I shall call these FIRP, and . . .
            >
            > > > > b) some "pass" outputs incorrectly reported as "fail" I shall call these PIRF
            >
            > > > >
            >
            > > > > Now, when I query with new data, I obviously want a small number of both FIRP and PIRF instances.
            >
            > > > > However it is much more important (in my particular application) to reduce the number of FIRP instances, than the PIRF ones.
            >
            > > > > In other words, the ratio FIRPs/PIRFs should be a minimum, but also the total of FIRPs + PIRFs should be as low as possible, for each 100 validation examples.
            >
            > > > >
            >
            > > > > Now to my question.
            >
            > > > > Is there any way I can change the training settings to minimize the number of FIRP cases (rather than both FIRP and PIRF)?
            >
            > > > > Also, how does my FIRP and PIRF terminology relate to the data in the Diagnostic Array ?
            >
            > > > >
            >
            > > > > I am using Easy-NN ver 15.0a
            >
            > > > > Any help would be much appreciated.
            >
            > > > >
            >
            > > > > Thanks,
            >
            > > > > Paul
            >
            > > > >
            >
            > > >
            >
            > >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            >
            > [Non-text portions of this message have been removed]
            >
          Your message has been successfully submitted and would be delivered to recipients shortly.