Loading ...
Sorry, an error occurred while loading the content.

Re: [hackers-il] Automatic Test Suites

Expand Messages
  • Omer Zak
    ... Where&when did I say the above? My position about tests in open source projects is that tests should be considered to be part of the source code, because
    Message 1 of 6 , Jan 3, 2003
    • 0 Attachment
      On Fri, 3 Jan 2003, Shlomi Fish wrote:

      > Now Chen asked me if I have prepared such automatic tests for Freecell
      > Solver, and I answered that I did not encounter it. OmerZ took the
      > opposite approach and claimed open-source developers should not bother
      > with automatic tests and should rely on their users for them.

      Where&when did I say the above?

      My position about tests in open source projects is that tests should be
      considered to be part of the source code, because they help certify that
      nothing unexpected was broken by an attempt to modify the software. In
      order to be confident at a modified version of complicated software, such
      a certification is sometimes necessary.
      --- Omer
      WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
    • Shlomi Fish
      ... http://groups.yahoo.com/group/hackers-il/message/1491 Although it could have been a joke... ... OK. Sorry for misinterpreting you. Regards, Shlomi Fish ...
      Message 2 of 6 , Jan 3, 2003
      • 0 Attachment
        On Fri, 3 Jan 2003, Omer Zak wrote:

        >
        > On Fri, 3 Jan 2003, Shlomi Fish wrote:
        >
        > > Now Chen asked me if I have prepared such automatic tests for Freecell
        > > Solver, and I answered that I did not encounter it. OmerZ took the
        > > opposite approach and claimed open-source developers should not bother
        > > with automatic tests and should rely on their users for them.
        >
        > Where&when did I say the above?
        >

        http://groups.yahoo.com/group/hackers-il/message/1491

        Although it could have been a joke...

        > My position about tests in open source projects is that tests should be
        > considered to be part of the source code, because they help certify that
        > nothing unexpected was broken by an attempt to modify the software.In
        > order to be confident at a modified version of complicated software, such
        > a certification is sometimes necessary.

        OK. Sorry for misinterpreting you.

        Regards,

        Shlomi Fish

        > --- Omer
        > WARNING TO SPAMMERS:at http://www.zak.co.il/spamwarning.html
        >
        >
        >
        > To unsubscribe from this group, send an email to:
        > hackers-il-unsubscribe@egroups.com
        >
        >
        >
        > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
        >
        >



        ----------------------------------------------------------------------
        Shlomi Fish shlomif@...
        Home Page: http://t2.technion.ac.il/~shlomif/
        Home E-mail: shlomif@...

        "Let's suppose you have a table with 2^n cups..."
        "Wait a second - is n a natural number?"
      • Omer Zak
        ... It was indeed a joke. The above message contained two contributions of mine to Shlomi Fish s The Rules of Open-Source Programming . ... WARNING TO
        Message 3 of 6 , Jan 3, 2003
        • 0 Attachment
          On Fri, 3 Jan 2003, Shlomi Fish wrote:

          > On Fri, 3 Jan 2003, Omer Zak wrote:
          >
          > >
          > > On Fri, 3 Jan 2003, Shlomi Fish wrote:
          > >
          > > > Now Chen asked me if I have prepared such automatic tests for Freecell
          > > > Solver, and I answered that I did not encounter it. OmerZ took the
          > > > opposite approach and claimed open-source developers should not bother
          > > > with automatic tests and should rely on their users for them.
          > >
          > > Where&when did I say the above?
          > >
          >
          > http://groups.yahoo.com/group/hackers-il/message/1491
          >
          > Although it could have been a joke...

          It was indeed a joke. The above message contained two contributions of
          mine to Shlomi Fish's "The Rules of Open-Source Programming".

          --- Omer
          WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
        • Shlomi Fish
          ... Well, then I may have taken the joke too seriously and believed it reflected your opinion. Naturally, some open-source projects can rely on their users to
          Message 4 of 6 , Jan 4, 2003
          • 0 Attachment
            On Fri, 3 Jan 2003, Omer Zak wrote:

            >
            > On Fri, 3 Jan 2003, Shlomi Fish wrote:
            >
            > > On Fri, 3 Jan 2003, Omer Zak wrote:
            > >
            > > >
            > > > On Fri, 3 Jan 2003, Shlomi Fish wrote:
            > > >
            > > > > Now Chen asked me if I have prepared such automatic tests for Freecell
            > > > > Solver, and I answered that I did not encounter it. OmerZ took the
            > > > > opposite approach and claimed open-source developers should not bother
            > > > > with automatic tests and should rely on their users for them.
            > > >
            > > > Where&when did I say the above?
            > > >
            > >
            > > http://groups.yahoo.com/group/hackers-il/message/1491
            > >
            > > Although it could have been a joke...
            >
            > It was indeed a joke.The above message contained two contributions of
            > mine to Shlomi Fish's "The Rules of Open-Source Programming".
            >

            Well, then I may have taken the joke too seriously and believed it
            reflected your opinion. Naturally, some open-source projects can rely on
            their users to locate bugs for them. This is especially true of a project
            which does not have very good security concerns, or a GUI one that is hard
            to test automatically.

            Still, if you can write test suites, than maybe you should, despite the
            fact that a large number of projects (including mine) don't bother to.

            Regards,

            Shlomi Fish

            > --- Omer
            > WARNING TO SPAMMERS:at http://www.zak.co.il/spamwarning.html
            >
            >
            >
            > To unsubscribe from this group, send an email to:
            > hackers-il-unsubscribe@egroups.com
            >
            >
            >
            > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
            >
            >



            ----------------------------------------------------------------------
            Shlomi Fish shlomif@...
            Home Page: http://t2.technion.ac.il/~shlomif/
            Home E-mail: shlomif@...

            "Let's suppose you have a table with 2^n cups..."
            "Wait a second - is n a natural number?"
          • guy keren
            ... i ve seen several good test suites, and i ve participated in writing 2 others. i distinguish between correctness tests and stress tests. correctness tests
            Message 5 of 6 , Jan 4, 2003
            • 0 Attachment
              On Fri, 3 Jan 2003, Shlomi Fish wrote:

              > My question is: what experience have you had with automatic test suites?
              > How much do they help identifying bugs and broken code? What kind of tests
              > would you recommend?

              i've seen several good test suites, and i've participated in writing 2
              others. i distinguish between correctness tests and stress tests.

              correctness tests deal with checking that a specific function is done
              properly (gives the expected reuslts). this is mostly useful when you
              re-write your code and want to make sure it didn't break, or when you add
              new features to a large feature-full project.

              stress tests deal with bombarding your code in some manner, and making
              sure it does not crash. this is useful both in early development stage,
              and when the project is mature.

              i've seen stress testing doing a lot of good, both for the developers, and
              for users. as a developer - it helps finding instability bugs, races and
              the like. its not somehting that comes instead of manual testing - but
              rather complements it.

              when a project is small, or does something small, it is rather easy to
              test it. when it becomes large, and contains a lot of features, manual
              tests simply don't cope well with the task. when i was working on large
              industry project, which was first only being tested manually, i saw that
              we got bugs re-crawling into the system (sometimes the same bugs,
              soemtimes new bugs with the same symptoms). after a while, the testing
              team started writing automatic tests for the code (it was driven by a GUI
              on Unix, so they used X-runner for the testing) - and eventually, a single
              machine was dedicated to run automatic regression tests on the latest
              'internal release' version of the code all the time - and this helped us
              find bugs faster. finding a bug close to when you create it is very
              helpful - since it usually lies in the new code you just added (thought
              someitmes its a bug in the old code, that the new code now exposed due to
              using the old code in new ways).

              thus, i think my (current) conclusion is that automatic tests should be
              developed along side the development of the software, and the ammount of
              effort spent on it, should be proportional to the size and complexity of
              the software, as well as to the maturity stage of the software (the more
              mature it is - the more tests you need in order to make sure it keeps
              ticking).

              --
              guy

              "For world domination - press 1,
              or dial 0, and please hold, for the creator." -- nob o. dy
            Your message has been successfully submitted and would be delivered to recipients shortly.