Loading ...
Sorry, an error occurred while loading the content.
 

Re: random number generator--again!

Expand Messages
  • suwailems
    ... Here is what I mean exactly : ifelse fix-random-seed = true [ do-sequence-as-in-v3.0 ] [ do-sequence-randomly ] end I hope it is clear?
    Message 1 of 30 , Sep 1, 2007
      > I'm afraid I don't understand this suggestion. What does "the
      >sequence shall be fixed and constant" mean, exactly?



      Here is what I mean "exactly":

      ifelse fix-random-seed = true
      [ do-sequence-as-in-v3.0 ]
      [ do-sequence-randomly ]
      end


      I hope it is clear?
    • David O'Sullivan
      Always fun keeping a thread alive... I agree with the NetLogo team (as represented here by Seth) that these issues are simply unavoidable - and very difficult!
      Message 2 of 30 , Sep 3, 2007
        Always fun keeping a thread alive...

        I agree with the NetLogo team (as represented here by Seth) that these
        issues are simply unavoidable - and very difficult! - in the end.

        It makes perfect sense to me for _ask_ to be random order by default,
        since this avoids problems of model artifacts due to execution order.
        I'm less sure about this for the values-from (or [reporter] of in v4)
        construct - see later.

        Once a decision has been made that ask should be random-order (and I
        guess Sami disagrees with this decision!) then there is no obvious way
        (to me) to avoid it applying to any situation where this construct occurs.



        If I understand Sami correctly, I think what he (she?) is after is
        something like the ability at the 'top' of a model to say something like

        set ask-order-fixed? true

        as a global. Having made this setting then ask would always return the
        agentset in the same order (perhaps the who order, or some other order,
        but always the same order). However, I'm not clear as to how this would
        work in a model where turtles are dying and being born?

        with-local-randomness isn't a bad workaround, particularly for all those
        values-from (or [reporter] of in v4) calls that are used simply to
        calculate statistics. However, for me it does raise the design question
        as to whether values-from (or in v4 [reporter] of) should necessarily
        return randomly ordered lists, and hence invoke the RNG, rather than
        fixed order lists. This is probably a metaphysical question though,
        with no right answer!



        Another option to think about would be an extension (using a java API
        such as colt) with named RNGs for different purposes. This might have
        reporters and procedures like:

        make-rng NAME ;; make a new named RNG
        set-seed NAME seed-value ;; set the seed of the named RNG
        random-float NAME 1 ;; get next float from RNG with the specified name

        This would still leave the issue of ask and values-from making 'hidden'
        calls on the built in RNG though.



        One other option (which I hesitate to mention!) is James Steiner's
        deeply cunning (and mildly disturbing!) rapid ordered iteration
        construct reported in this message:

        http://groups.yahoo.com/group/netlogo-users/message/6599

        I'm not sure I want to go there though...

        David
      • suwailems
        Thank you David for your comment. Just for record, I have raised the issue of order of execution in 2004 (message # 2071), quoting Epstein and Axtell from
        Message 3 of 30 , Sep 4, 2007
          Thank you David for your comment.

          Just for record, I have raised the issue of order of execution in
          2004 (message # 2071), quoting Epstein and Axtell from their book:
          Growing Artificail Societies, on why random order is necessary for
          robustness of results. As you know, at that time NetLogo was
          adopting the fixed order procedure. So I cannot disagree with a
          point I have raised my self three years ago!

          My point, I don't mind repeating, is to give the user the option
          either to have random order or fixed order. I just cannot see how
          this is too difficult? It was fixed up to 3.0. The question you
          raised about turtles dying, will be treated the same way as it was
          in v3.0 IF the option of fixing the order was on.

          The default should be to have a random order to assure robustness of
          results. But for comparative statics, i.e. comparing two versions of
          the same model with one or two different conditions , you have to
          fix other variables. If the order is not fixed, you will be getting
          differing results that you cannot identify its source: Is it the
          order of agents, or is it the conditions imposed (or relaxed)?

          It seems to me very intuitive to have the two options (random and
          fixed order), and quite obvious how to implement: if the option of
          fixed order is on, do it as in v3.0, else do it randomly as it is
          now. You think this is very difficult?

          Best regards

          Your brother: Sami




          --- In netlogo-users@yahoogroups.com, David O'Sullivan
          <d.osullivan@...> wrote:
          >
          > Always fun keeping a thread alive...
          >
          > I agree with the NetLogo team (as represented here by Seth) that
          these
          > issues are simply unavoidable - and very difficult! - in the end.
          >
          > It makes perfect sense to me for _ask_ to be random order by
          default,
          > since this avoids problems of model artifacts due to execution
          order.
          > I'm less sure about this for the values-from (or [reporter] of in
          v4)
          > construct - see later.
          >
          > Once a decision has been made that ask should be random-order (and
          I
          > guess Sami disagrees with this decision!) then there is no obvious
          way
          > (to me) to avoid it applying to any situation where this construct
          occurs.
          >
          >
          >
          > If I understand Sami correctly, I think what he (she?) is after is
          > something like the ability at the 'top' of a model to say
          something like
          >
          > set ask-order-fixed? true
          >
          > as a global. Having made this setting then ask would always
          return the
          > agentset in the same order (perhaps the who order, or some other
          order,
          > but always the same order). However, I'm not clear as to how this
          would
          > work in a model where turtles are dying and being born?
          >
          > with-local-randomness isn't a bad workaround, particularly for all
          those
          > values-from (or [reporter] of in v4) calls that are used simply to
          > calculate statistics. However, for me it does raise the design
          question
          > as to whether values-from (or in v4 [reporter] of) should
          necessarily
          > return randomly ordered lists, and hence invoke the RNG, rather
          than
          > fixed order lists. This is probably a metaphysical question
          though,
          > with no right answer!
          >
          >
          >
          > Another option to think about would be an extension (using a java
          API
          > such as colt) with named RNGs for different purposes. This might
          have
          > reporters and procedures like:
          >
          > make-rng NAME ;; make a new named RNG
          > set-seed NAME seed-value ;; set the seed of the named RNG
          > random-float NAME 1 ;; get next float from RNG with the specified
          name
          >
          > This would still leave the issue of ask and values-from
          making 'hidden'
          > calls on the built in RNG though.
          >
          >
          >
          > One other option (which I hesitate to mention!) is James Steiner's
          > deeply cunning (and mildly disturbing!) rapid ordered iteration
          > construct reported in this message:
          >
          > http://groups.yahoo.com/group/netlogo-users/message/6599
          >
          > I'm not sure I want to go there though...
          >
          > David
          >
        • Ken Kahn
          Sami wrote: The default should be to have a random order to assure robustness of results. But for comparative statics, i.e. comparing two versions of the same
          Message 4 of 30 , Sep 4, 2007
            Sami wrote:
             
            The default should be to have a random order to assure robustness of
            results. But for comparative statics, i.e. comparing two versions of
            the same model with one or two different conditions , you have to
            fix other variables. If the order is not fixed, you will be getting
            differing results that you cannot identify its source: Is it the
            order of agents, or is it the conditions imposed (or relaxed)?

            ===============
             
            If your model uses random numbers then I don't see how you can identify the source even if the order of agents is fixed. Perhaps due to different conditions one of the agents needs a different number of random numbers than with the initial conditions then all agents after that will be using different random numbers.
             
            Seems to me the only solution is to run your model with random ordering of agents MANY times and look at the statistics to determine if you are getting significantly different results due to different conditions.
             
            Best,
             
            -ken
          • suwailems
            To see how sequence order of agents may give unexpected results, consider the following simple model: You have two variables 1. x, which is purely random. 2. y
            Message 5 of 30 , Sep 5, 2007
              To see how sequence order of agents may give unexpected results,
              consider the following simple model:

              You have two variables
              1. x, which is purely random.
              2. y = f(x),

              x is an exogenous, independent variable, while y is endogenous,
              dependent variable.

              Now fix the seed number, and run the model 100 times with
              calculating y, and run the same model with the same seed but without
              calculating y.

              If y is calculated with a separate ask, you will get two different
              results of the average of x. That is, just because we are not
              calculating y, x changes. This is very strange since x is exogenous
              by construction, and thus it shouldn't be sensitive to whether we
              calculate y or not. The order sequence inadvertently creates a
              spurious dependency of x on y, so that the supposedly exogenous
              variable (x) is not exogenous any more!

              If you just think of it, you immediately realize how results now
              become very sensitive to number of ask commands, which makes the
              results not robust as desired. The mechanism of implementing the
              randomized order, in other words, might defeat its own purpose,
              namely to have robust results!

              Sami
            • Seth Tisue
              ... suwailems If y is calculated with a separate ask, you will get two suwailems different results of the average of x. That is, just because suwailems we
              Message 6 of 30 , Sep 5, 2007
                >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:

                suwailems> If y is calculated with a separate ask, you will get two
                suwailems> different results of the average of x. That is, just because
                suwailems> we are not calculating y, x changes. This is very strange
                suwailems> since x is exogenous by construction, and thus it shouldn't
                suwailems> be sensitive to whether we calculate y or not. The order
                suwailems> sequence inadvertently creates a spurious dependency of x on
                suwailems> y, so that the supposedly exogenous variable (x) is not
                suwailems> exogenous any more!

                I would suggest wrapping the calculation of Y in WITH-LOCAL-RANDOMNESS.
                The scenario you describe is *exactly* why we added it to the language.

                I think this solution is decidedly preferable to the solution you have
                suggested, which is to make a global change to the entire model where
                you fix agentset order everywhere in the model for the entire run.
                That's an enormous change to the model! WITH-LOCAL-RANDOMNESS lets you
                calculate y without affecting your runs and without making a drastic
                change to the way the whole model works.

                I don't think I have much left to say on this topic, except to second
                what Ken Kahn wrote:

                Ken> If your model uses random numbers then I don't see how you can
                Ken> identify the source even if the order of agents is fixed. Perhaps
                Ken> due to different conditions one of the agents needs a different
                Ken> number of random numbers than with the initial conditions then all
                Ken> agents after that will be using different random numbers. Seems
                Ken> to me the only solution is to run your model with random ordering
                Ken> of agents MANY times and look at the statistics to determine if
                Ken> you are getting significantly different results due to different
                Ken> conditions.

                --
                Seth Tisue / http://tisue.net
                lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
              • David O'Sullivan
                I agree with Seth (and with Ken) here (and Seth, you ve been very thoughtful in your replies on this topic - I agree, it s interesting stuff!). First time I
                Message 7 of 30 , Sep 6, 2007
                  I agree with Seth (and with Ken) here (and Seth, you've been very
                  thoughtful in your replies on this topic - I agree, it's interesting
                  stuff!).

                  First time I noticed some of the issues was in 3.1.1. when I was having
                  trouble replicating results and found that a monitor that used an ask or
                  values-from call (or both, I can't remember the details) was throwing
                  things off and preventing my careful use of random seeds from giving me
                  identical runs from one to the next. This was while I was debugging a
                  problem, where repeatability was important for replicating a problem,
                  and it took a while for me to find the root cause (the 'hidden' random
                  call in the monitor code).

                  The with-local-randomness construct would have been perfect in that
                  situation, and I'll be using it from now on in all the
                  reporting/output/stats calculating parts of my models. Having looked
                  into it further, this is a great addition to the language!

                  [Of course, in 4.0, monitor code uses its own RNG (I believe?) so this
                  problem wouldn't have arisen at all...]

                  I'm glad to see everyone agreeing that random-order ask is important - I
                  recall having to go to a great deal of trouble in a Repast model I built
                  a while back to force exactly this behaviour - it's nice in NetLogo not
                  to have to think about it.

                  Having said that, I don't think I care whether values-from (or [value]
                  of) returns random order lists. I don't really expect that function to
                  be randomised, since the result is a list which is intrinsically an
                  ordered entity. Further, more often than not the next thing I will do
                  with such a list is to calculate a mean or sum or whatever, and order is
                  not a concern. If what I plan to do is order-dependent, then I probably
                  have to sort the list anyway, under either scheme. And might there even
                  be some sort of performance benefit of not doing the 'hidden' shuffle?

                  On the other hand, this is not a big enough concern for me to worry
                  about whether such a change gets made!

                  David
                • Seth Tisue
                  ... David [Of course, in 4.0, monitor code uses its own RNG (I believe?) That s correct. David Further, more often than not the next thing I will do with
                  Message 8 of 30 , Sep 6, 2007
                    >>>>> "David" == David O'Sullivan <d.osullivan@...> writes:

                    David> [Of course, in 4.0, monitor code uses its own RNG (I believe?)

                    That's correct.

                    David> Further, more often than not the next thing I will do with such
                    David> a list is to calculate a mean or sum or whatever, and order is
                    David> not a concern. If what I plan to do is order-dependent, then I
                    David> probably have to sort the list anyway, under either scheme. And
                    David> might there even be some sort of performance benefit of not
                    David> doing the 'hidden' shuffle?

                    In general, the performance cost of shuffling is surprisingly low. When
                    we were first experimenting with making agentsets automatically be
                    shuffled, we were worried that models would slow down, but in practice
                    we found it typically only made a percent or so difference.

                    NetLogo is smart enough in some cases to avoid unnecessary shuffles, for
                    efficiency's sake. For example, as James Steiner discovered, WITH
                    doesn't bother to shuffle. Nor does ALL?. There are even some special
                    combinations of primitives that don't shuffle, such as OF used in
                    direction combination with MEAN, SUM, MIN, or MAX. The assumption we're
                    making is that the reporters you use these primitives with don't have
                    side effects.

                    --
                    Seth Tisue / http://tisue.net
                    lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                  • suwailems
                    Thank you Seith for elaboration. I did try wrapping cal-y with WITH-LOCAL-RANDOMNESS, and results were the same. But I was wondering, why did the average of x
                    Message 9 of 30 , Sep 8, 2007
                      Thank you Seith for elaboration.

                      I did try wrapping cal-y with WITH-LOCAL-RANDOMNESS, and results were
                      the same.

                      But I was wondering, why did the average of x change when the
                      average is independent of the order of agents?
                    • Seth Tisue
                      ... suwailems Thank you Seith for elaboration. I did try wrapping cal-y suwailems with WITH-LOCAL-RANDOMNESS, and results were the same. suwailems But I
                      Message 10 of 30 , Sep 8, 2007
                        >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:

                        suwailems> Thank you Seith for elaboration. I did try wrapping cal-y
                        suwailems> with WITH-LOCAL-RANDOMNESS, and results were the same.
                        suwailems> But I was wondering, why did the average of x change when
                        suwailems> the average is independent of the order of agents?

                        Sorry, I'm not understanding the question. (Maybe you could send some
                        actual NetLogo code that shows what you mean...?)

                        --
                        Seth Tisue / http://tisue.net
                        lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                      • suwailems
                        Please ignore previous incomplete post! Here is a simple example: globals [time] patches-own [x y] to setup clear-all random-seed 01234 end to gen-x ask
                        Message 11 of 30 , Sep 9, 2007
                          Please ignore previous incomplete post!

                          Here is a simple example:


                          globals [time]

                          patches-own [x y]

                          to setup
                          clear-all
                          random-seed 01234
                          end

                          to gen-x
                          ask patches [set x random 100]
                          end

                          to cal-y
                          if do-cal-y = true ; do-cal-y is set through a switch button.
                          [ ask patches [set y x / 10] ]
                          end



                          to go
                          set time time + 1
                          gen-x
                          cal-y
                          if time = 10 [stop]
                          end


                          Now make a monitor of "mean [x] of patches", and run the model once
                          with the switch on, and once when it is off. You will get two
                          different means of x.

                          Quick fix: I tried replacing cal-y by the following:

                          to cal-y
                          ifelse do-cal-y = true
                          [ ask patches [set y x / 10] ]
                          [ ask patches [] ]
                          end


                          The empty ask patches [] command makes the mean x the same whether
                          the switch is on of off.

                          Sami
                        • Seth Tisue
                          ... suwailems Now make a monitor of mean [x] of patches , and run the suwailems model once with the switch on, and once when it is off. You suwailems will
                          Message 12 of 30 , Sep 9, 2007
                            >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:

                            suwailems> Now make a monitor of "mean [x] of patches", and run the
                            suwailems> model once with the switch on, and once when it is off. You
                            suwailems> will get two different means of x.

                            In my testing in NetLogo 4.0beta7, wrapping cal-y in
                            WITH-LOCAL-RANDOMNESS makes the difference go away, as expected.

                            What's your question...?

                            --
                            Seth Tisue / http://tisue.net
                            lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                          • suwailems
                            ... The question is why did the mean of x change in the first place, i.e. with out the W-L-R? The mean is supposed to be indpendent of the order of agents, so
                            Message 13 of 30 , Sep 9, 2007
                              > In my testing in NetLogo 4.0beta7, wrapping cal-y in
                              > WITH-LOCAL-RANDOMNESS makes the difference go away, as expected.
                              >
                              > What's your question...?
                              >

                              The question is why did the mean of x change in the first place, i.e.
                              with out the W-L-R? The mean is supposed to be indpendent of the order
                              of agents, so whether the order is fixed or randomized, the mean should
                              be the same. But we are getting different results of the mean dependeng
                              on the order. Why is that?
                            • Seth Tisue
                              ... suwailems The question is why did the mean of x change in the first place, i.e. suwailems with out the W-L-R? The mean is supposed to be indpendent of
                              Message 14 of 30 , Sep 10, 2007
                                >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:

                                >> In my testing in NetLogo 4.0beta7, wrapping cal-y in
                                >> WITH-LOCAL-RANDOMNESS makes the difference go away, as expected.
                                >>
                                >> What's your question...?

                                suwailems> The question is why did the mean of x change in the first place, i.e.
                                suwailems> with out the W-L-R? The mean is supposed to be indpendent of
                                suwailems> the order of agents, so whether the order is fixed or
                                suwailems> randomized, the mean should be the same. But we are getting
                                suwailems> different results of the mean dependeng on the order. Why is
                                suwailems> that?

                                The means are different because you're averaging a completely different
                                set of numbers...!

                                After calling GO once, the means are the same. But if you don't use
                                WITH-LOCAL-RANDOMNESS in cal-y, then the state of the random number
                                generator is different, so the second time you call GO the gen-x
                                procedure generates different values for x, and thus the mean is
                                different.

                                --
                                Seth Tisue / http://tisue.net
                                lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                              • suwailems
                                ... Thank you for elaboration. So I understand that all random numbers in a model are controlled by a single generator? If this is so, this means that, if X, Y
                                Message 15 of 30 , Sep 10, 2007
                                  > After calling GO once, the means are the same. But if you don't use
                                  > WITH-LOCAL-RANDOMNESS in cal-y, then the state of the random number
                                  > generator is different, so the second time you call GO the gen-x
                                  > procedure generates different values for x, and thus the mean is
                                  > different.
                                  >

                                  Thank you for elaboration.

                                  So I understand that all random numbers in a model are controlled by a
                                  single generator? If this is so, this means that, if X, Y and Z are
                                  random numbers in a model, then not generating Z will affect next draws
                                  of X and Y. I can see why W-L-R is important. But is it possible to
                                  keep the random numbers independent in the first place?
                                • Seth Tisue
                                  ... suwailems So I understand that all random numbers in a model are suwailems controlled by a single generator? Yes. suwailems If this is so, this means
                                  Message 16 of 30 , Sep 12, 2007
                                    >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:

                                    suwailems> So I understand that all random numbers in a model are
                                    suwailems> controlled by a single generator?

                                    Yes.

                                    suwailems> If this is so, this means
                                    suwailems> that, if X, Y and Z are random numbers in a model, then not
                                    suwailems> generating Z will affect next draws of X and Y. I can see
                                    suwailems> why W-L-R is important. But is it possible to keep the
                                    suwailems> random numbers independent in the first place?

                                    No.

                                    The random seed for a run is really only useful for one thing, namely
                                    reproducing the same run later. And when I say "the same run" I mean
                                    *exactly* the same run: the same exact code running in the same exact
                                    version of NetLogo.

                                    --
                                    Seth Tisue / http://tisue.net
                                    lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                                  • abm0000abm
                                    Hello All, I have been working in NetLogo for a few months now. I need to do some comparative statics and was trying to figure out a way to set random seed. It
                                    Message 17 of 30 , Jul 22, 2011
                                      Hello All,

                                      I have been working in NetLogo for a few months now. I need to do some comparative statics and was trying to figure out a way to set random seed. It appeared that _random-seed_ should be the straightforward solution but, apparently, it isn't.

                                      I use:

                                      "to setup
                                      ca
                                      setup-globals
                                      setup-people
                                      random-seed 1000
                                      end"

                                      for setup with more code for setups and _go_. I run the exact same model as multiple (7) instances on a 4-core hyperthreading machine. I should get the same result from all seven runs (instances) but I do not.

                                      I tried to find an answer in the archives but could not. Any help would be deeply appreciated.

                                      Regards,

                                      Ajay



                                      --- In netlogo-users@yahoogroups.com, Seth Tisue <seth@...> wrote:
                                      >
                                      > >>>>> "suwailems" == suwailems <no_reply@yahoogroups.com> writes:
                                      >
                                      > suwailems> So I understand that all random numbers in a model are
                                      > suwailems> controlled by a single generator?
                                      >
                                      > Yes.
                                      >
                                      > suwailems> If this is so, this means
                                      > suwailems> that, if X, Y and Z are random numbers in a model, then not
                                      > suwailems> generating Z will affect next draws of X and Y. I can see
                                      > suwailems> why W-L-R is important. But is it possible to keep the
                                      > suwailems> random numbers independent in the first place?
                                      >
                                      > No.
                                      >
                                      > The random seed for a run is really only useful for one thing, namely
                                      > reproducing the same run later. And when I say "the same run" I mean
                                      > *exactly* the same run: the same exact code running in the same exact
                                      > version of NetLogo.
                                      >
                                      > --
                                      > Seth Tisue / http://tisue.net
                                      > lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                                      >
                                    • Seth Tisue
                                      ... abm0000abm I use: abm0000abm to setup ca setup-globals setup-people random-seed 1000 abm0000abm end You should set the random seed at the start of
                                      Message 18 of 30 , Jul 23, 2011
                                        >>>>> "abm0000abm" == abm0000abm <usedaccount1@...> writes:

                                        abm0000abm> I use:
                                        abm0000abm> "to setup ca setup-globals setup-people random-seed 1000
                                        abm0000abm> end"

                                        You should set the random seed at the start of setup, not the end.
                                        Setting it at the end isn't meaningful if there's anything random in the
                                        code preceding. And there almost certainly is something random; a
                                        myriad of NetLogo primitives involve random choices.

                                        --
                                        Seth Tisue | Northwestern University | http://tisue.net
                                        lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
                                      Your message has been successfully submitted and would be delivered to recipients shortly.