Loading ...
Sorry, an error occurred while loading the content.

Re: [junit] Re: Alphabetizing test method run order?

Expand Messages
  • David Saff
    ... That s a fair request. Actually, I think the code has evolved to the point that adding a feature to ParentRunner to support this would be remarkably easy:
    Message 1 of 11 , Sep 22, 2011
    View Source
    • 0 Attachment
      On Thu, Sep 22, 2011 at 9:08 AM, mjfarwell <matthew@...> wrote:
      >
      >
      > --- In junit@yahoogroups.com, Kevin Cooney <kcooney@...> wrote:
      >>
      >> On Tue, Sep 20, 2011 at 3:07 AM, Dominik Dorn <dorn@...> wrote:
      >>
      >> > **
      >> >
      >> >
      >> > Hmm... I think it would be even better to _ensure_ that the test cases are
      >> > run
      >> > in a random order but provide a summary at the end of the testrun where the
      >> > order gets printed.
      >> >
      >>
      >> -10
      >>
      >> A number of projects I worked on used to do this. While this may initially
      >> seem like a good idea, it was problematic for us for a number of reasons.
      >>
      >> If the tests are run in a random order, then a small change to a test or a
      >> class under test could cause a completely unrelated test to fail. Many
      >> developers will assume that it is their change that caused the problem,
      >> leading them in a wild goose chase trying to see which of their changes
      >> caused the problems and why, and when they realize it is due to the test
      >> ordering, they need to do a big context switch back to their original
      >> task. This would be made worse by the fact that simply rerunning the tests
      >> would cause them to pass again (or worse, cause a different test to fail).
      >>
      >> Note that I don't think developers should be writing tests where the outcome
      >> depends on the ordering that the test methods were run. I just don't think
      >> we should assume that fixing the tests with this problem is necessarily as
      >> important as what the developer is trying to do.
      >>
      >> Making the test order random by default also does not necessarily make it
      >> easier to track down problems with ordering of tests, since if you ran the
      >> failing test suite again, you would get a different run order. JUnit could
      >> print out the random seed used for that particular run, but then developers
      >> would have to figure out how to pass the previous random seed back to the
      >> test (how to do this would depend on which IDE or build system they used).
      >
      > +10.
      >
      >> If developers want the ordering to be shuffled, they should use a custom
      >> runner that shuffles the tests. We used to do this in our JUnit3 tests with
      >> a custom subclass of TestSuite. It used a random seed based on the
      >> System.currentTimeMillis() at midnight, so if a particular ordering caused
      >> the tests to fail, it was reproducible for the rest of the day.
      >> Unfortunately, developers often assumed that the breakage was caused by one
      >> of the changes submitted since the most recent green run of
      >> the continuous build.
      >
      > I think this is a very good point. The ordering absolutely needs to be deterministic by default, so any problems are reproducible easily.
      >
      >> Personally, I don't need the test ordering to be defined, but I need it to
      >> be deterministic. If I build the same code with the same JDK and run it in
      >> the same JRE on the same kernel on a different day, the tests should run in
      >> the same order. We often re-run tests that failed earlier in the day, using
      >> the same source code, to try to determine which failures were due to flaky
      >> tests. Having the tests run in a different order would make it much more
      >> difficult to do this analysis.
      >
      > I do use the ordering of junit tests, not because the tests are inter-dependent, but simply because I have slow tests and fast tests. I use junit for integration tests, so I run a batch from junit, and look at the results of the batch. Some tests have small datasets (10-15 seconds) and some have large datasets (10 minutes). I put the small ones first so I can either stop the test if the first fails, or do something else for ten minutes.
      >
      > If I have to run these tests using a different Runner, then that's fine, but I would rather not have to change anything (major).
      >
      >> I suggest that JUnit4 order the tests by the hash code of the test method
      >> name. If there is a tie, JUnit could compare the test names using the
      >> natural ordering defined by String.compareTo(String) (possibly in reverse
      >> :-)
      >
      > I would suggest we use the principle of least surprise here. If the hashCode is used as an ordering, yes it is deterministic, but it will seem random. If you use the test name, it is less surprising. People are used to tests being run in bytecode order.
      >
      > If you change the order then at least change it so that if people have problems due to ordering they can easily change the order. Yes, I know that their tests are broken, but they probably do too. If the order is based upon the hashCode, I have to either run that test with a custom Runner (and make sure that it's usable in Eclipse, Jenkins, maven, etc. etc.)[*]. If the order is simply the name, then all I have to do to 'fix' my problem is change the name.

      That's a fair request. Actually, I think the code has evolved to the
      point that adding a feature to ParentRunner to support this would be
      remarkably easy:

      @RunWith(Suite.class)
      @SuiteClasses({...})
      public class MySuite {
      @SortWith public static Sorter alpha = Sorters.ALPHABETICAL;
      @FilterWith public static Filter category = new
      CategoryFilter(MyCategory.class);
      }

      Anyone want to take a crack at it?

      David Saff

      >
      > Matthew Farwell.
      >
      > [*] Or change the hashCode. For instance, I could have two tests:
      >
      > public void testStuff()
      > public void testMoreStuff()
      >
      > These run in order as above. If the hashCode were used to order them, then we get testMoreStuff() then testStuff(). So to keep the original order we have to change testMoreStuff to testMoreStuffTextToIncreaseHashCode() :-)
      >
      >
      >
      > ------------------------------------
      >
      > Yahoo! Groups Links
      >
      >
      >
      >
    • mjfarwell
      Hi, I ve submitted a pull request for this https://github.com/KentBeck/junit/pull/386. @SortMethodsWith(MethodSorters.NAME_ASC) public class MyTest { } There
      Message 2 of 11 , Feb 24, 2012
      View Source
      • 0 Attachment
        Hi,

        I've submitted a pull request for this https://github.com/KentBeck/junit/pull/386.

        @SortMethodsWith(MethodSorters.NAME_ASC)
        public class MyTest {
        }


        There are four possibilities for the MethodSorters:

        MethodSorters.DEFAULT: the default value, deterministic, but not predictable
        MethodSorters.JVM: the order in which the tests are returned by the JVM, i.e. there is no sorting done
        MethodSorters.NAME_ASC: sorted in order of method name, ascending
        MethodSorters.NAME_DESC: sorter in order of method name, descending

        I would love to have feedback on this proposal.

        Thanks.

        Matthew Farwell.

        --- In junit@yahoogroups.com, David Saff <david@...> wrote:
        >
        > On Thu, Sep 22, 2011 at 9:08 AM, mjfarwell <matthew@...> wrote:
        > > If you change the order then at least change it so that if people have problems due to ordering they can easily change the order. Yes, I know that their tests are broken, but they probably do too. If the order is based upon the hashCode, I have to either run that test with a custom Runner (and make sure that it's usable in Eclipse, Jenkins, maven, etc. etc.)[*]. If the order is simply the name, then all I have to do to 'fix' my problem is change the name.
        >
        > That's a fair request. Actually, I think the code has evolved to the
        > point that adding a feature to ParentRunner to support this would be
        > remarkably easy:
        >
        > @RunWith(Suite.class)
        > @SuiteClasses({...})
        > public class MySuite {
        > @SortWith public static Sorter alpha = Sorters.ALPHABETICAL;
        > @FilterWith public static Filter category = new
        > CategoryFilter(MyCategory.class);
        > }
        >
        > Anyone want to take a crack at it?
        >
        > David Saff
      Your message has been successfully submitted and would be delivered to recipients shortly.