Loading ...
Sorry, an error occurred while loading the content.

Re: Runner for maven

Expand Messages
  • kristian@rosenvold.com
    Surefire is maven s default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I d do in my own tests. If the user decides
    Message 1 of 12 , Jan 5, 2012
    View Source
    • 0 Attachment
      Surefire is maven's default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I'd do in my own tests. If the user decides to use loggers or system out is not really something we want to dictate.

      As a humoristic example, let me mention the issue we had on this guy spawning threads from a test that were producing console output for the entire test run that was collected by a JVM shutdown hook.

      Kristian


      --- In junit@yahoogroups.com, "Georg" <le_garcon_enerve@...> wrote:
      >
      > Dear Kristian,
      >
      > My first thought - and I hope not to sound arrogant - would be to
      > - to remove the the output altogether (who reads it anyhow once the test is run successfully the first time ;-); if one needs to read the output of the test runs, what's the point of unit tests?!).
      > - or to convert it into assertions (they also work nicely in methods not marked with @Test).
      >
      > If you still think you need the output, why not use a logger (log4j), which adds all the information you need? Refactoring should be very easy.
      >
      > As I'm starting to learn and play with AspectJ, I'd try that (but would not bet money on this being a good idea). This would allow to intercept calls to System.out.print() and alike, and tweak it print the argument(s) of the print()-statement augmented with the class to which the calling method belongs.
      >
      >
      >
      >
      > --- In junit@yahoogroups.com, kristian@ wrote:
      > >
      > > Sorry, I was doing just what I sometimes dislike of our own users ;)
      > >
      > > The real problem I want to solve is associating console output from a running thread to a given test-case (for parallel runs). We have this fairly well covered already and we manage to capture everything that happens within a given test-case for that test-case.
      > >
      > > The real problem is that the RunListener API does not demarcate test-classes properly, so I am unable to distinguish console output happening in @AfterClass of one test with @BeforeClass on the test-class on the same thread.
      > >
      > > This problem has been in the corner of my eye for quite some time and I've been trying to find a decent angle to solve it from. I promise not to do anything that is frowned upon, which is why I asked ;)
      > >
      > > Kristian
      > >
      > >
      > > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
      > > >
      > > > Kristian,
      > > >
      > > > In reading your original mail, I was also confused as to what you wanted to
      > > > achieve. Could you expand a little please?
      > > >
      > > > Thanks.
      > > >
      > > > Matthew Farwell.
      > > >
      > > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a écrit :
      > > >
      > > > > **
      > > > >
      > > > > Hi Kristian
      > > > >
      > > > > Do you really want a transition from RunListener?
      > > > >
      > > > > To an outside observer - a JUnit Runner "describes" its test-executions
      > > > > and reports their outcome to the RunNotifier, which forward the info to
      > > > > RunListener, which is the interface that 3rd-party test-running
      > > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
      > > > >
      > > > > I have seen IDE-project test-extensions try to go beyond
      > > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
      > > > > test-instance creation/setup, method-executions etc) and the result is
      > > > > something I don't like much. It works for simple cases but with customized
      > > > > Runner-classes and new JUnit-features (e.g. the introduction of
      > > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
      > > > > (i.e. the developers) from taking full advantage of JUnit!
      > > > >
      > > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
      > > > > JUnit-integration wonderfully, as far as I know, so your post worries me a
      > > > > little. What is it that you want to accomplish by moving away from the
      > > > > successful(!) "Junitcore+RunListener based approach"?
      > > > >
      > > > > Henrik
      > > > >
      > > > >
      > > > > --- In junit@yahoogroups.com, kristian@ wrote:
      > > > > >
      > > > > > Hi,
      > > > > >
      > > > > > I maintain the junit provider for maven-surefire, and I'm looking into
      > > > > transitioning from a Junitcore+RunListener based approach to an approach
      > > > > based on something like BlockJUnit4ClassRunner.
      > > > > >
      > > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
      > > > > client code itself may need to do that, so I'd somehow need to put "my"
      > > > > > Runner in front of the client-supplied one, maybe something proxy like.
      > > > > >
      > > > > > Anyone have any tips on this, maybe some existing code that does this ?
      > > > > >
      > > > > > Kristian
      > > > > >
      > > > >
      > > > >
      > > > >
      > > >
      > > >
      > > > [Non-text portions of this message have been removed]
      > > >
      > >
      >
    • Dawid Weiss
      ... I was thinking about ANT actually, not maven (surefire). I admit we also needed some extra control over suite allocation scheduling and more detailed
      Message 2 of 12 , Jan 5, 2012
      View Source
      • 0 Attachment
        > maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.

        I was thinking about ANT actually, not maven (surefire). I admit we
        also needed some extra control over suite allocation scheduling and
        more detailed reporting, so surefire wasn't a perfect fit for us. It
        is a great tool though, I agree with that.

        Dawid
      • Dawid Weiss
        Oh, I also didn t mention one more thing -- if you have tests that fork subthreads which in turn write to System.out/err then you will also need a smarter way
        Message 3 of 12 , Jan 5, 2012
        View Source
        • 0 Attachment
          Oh, I also didn't mention one more thing -- if you have tests that
          fork subthreads which in turn write to System.out/err then you will
          also need a smarter way to track which suite a thread belongs to.
          This can be done with a parent thread group (in most cases works) but
          like you said -- I think it's just a pain and there is no way to
          implement it correctly for all possible use cases.

          Dawid

          On Thu, Jan 5, 2012 at 12:52 PM, Dawid Weiss <dawid.weiss@...> wrote:
          >> maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.
          >
          > I was thinking about ANT actually, not maven (surefire). I admit we
          > also needed some extra control over suite allocation scheduling and
          > more detailed reporting, so surefire wasn't a perfect fit for us. It
          > is a great tool though, I agree with that.
          >
          > Dawid
        • Esko Luontola
          I m already working on something like that. I m writing a new test runner called Jumi http://jumi.fi/ which will have native support for running tests in
          Message 4 of 12 , Jan 6, 2012
          View Source
          • 0 Attachment
            I'm already working on something like that. I'm writing a new test
            runner called Jumi http://jumi.fi/ which will have native support for
            running tests in parallel in the same JVM, and it take care of capturing
            the System.out/err of individual tests.

            Jumi will do the output capturing by replacing System.out/err with its
            own implementation, which saves all print events (the same way it saves
            test lifecycle events), synchronizes both out and err (I think all
            current IDEs and build tools fail this:
            https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
            and also takes care of capturing the output of threads spawned by the
            test thread. For long-lived threads, such as AWT event dispatch thread,
            I will also try to figure out something (I'll timestamp all print
            events, so it should be possible to figure out roughly which test
            printed things in the AWT thread).

            The output capturing is at the top of my TODO list, so I will have it
            implemented after a couple of productive afternoons when I have time to
            work on it. Soon after that I'll make the first alpha release.

            I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
            Surefire have some overlap, for example both of them take care of
            classloading and starting a JVM process. One option is to use Surefire's
            classloading and use only the core runner parts of Jumi. Otherwise Jumi
            would need information that what are the dependency JARs, what are the
            current project's classes, maybe also that which of them are test and
            production classes (Jumi will do caching of the dependencies' class
            loaders to speed up the tests). That would require changes to Surefire,
            or then a completely new Maven plugin.


            kristian@... wrote on 4.1.2012 8:51:
            > Sorry, I was doing just what I sometimes dislike of our own users ;)
            >
            > The real problem I want to solve is associating console output from a
            > running thread to a given test-case (for parallel runs). We have this
            > fairly well covered already and we manage to capture everything that
            > happens within a given test-case for that test-case.
            >
            > The real problem is that the RunListener API does not demarcate
            > test-classes properly, so I am unable to distinguish console output
            > happening in @AfterClass of one test with @BeforeClass on the test-class
            > on the same thread.
            >
            > This problem has been in the corner of my eye for quite some time and
            > I've been trying to find a decent angle to solve it from. I promise not
            > to do anything that is frowned upon, which is why I asked ;)
            >
            > Kristian
            >

            --
            Esko Luontola
            www.orfjackal.net
          • Dawid Weiss
            You may want to look into RandomizedRunner/JUnit4 task -- it has most of the things you mentioned (synchronized i/o events with reporting, capturing runaway
            Message 5 of 12 , Jan 6, 2012
            View Source
            • 0 Attachment
              You may want to look into RandomizedRunner/JUnit4 task -- it has most
              of the things you mentioned (synchronized i/o events with reporting,
              capturing runaway threads and the like).

              The code is here:
              https://github.com/carrotsearch/randomizedtesting/

              I have a talk about randomized testing which includes the above
              concepts at Lucene EuroCon Barcelona, the video is here:
              http://vimeo.com/32087114

              Dawid

              On Fri, Jan 6, 2012 at 1:29 PM, Esko Luontola <esko.luontola@...> wrote:
              >
              >
              >
              > I'm already working on something like that. I'm writing a new test
              > runner called Jumi http://jumi.fi/ which will have native support for
              > running tests in parallel in the same JVM, and it take care of capturing
              > the System.out/err of individual tests.
              >
              > Jumi will do the output capturing by replacing System.out/err with its
              > own implementation, which saves all print events (the same way it saves
              > test lifecycle events), synchronizes both out and err (I think all
              > current IDEs and build tools fail this:
              > https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
              > and also takes care of capturing the output of threads spawned by the
              > test thread. For long-lived threads, such as AWT event dispatch thread,
              > I will also try to figure out something (I'll timestamp all print
              > events, so it should be possible to figure out roughly which test
              > printed things in the AWT thread).
              >
              > The output capturing is at the top of my TODO list, so I will have it
              > implemented after a couple of productive afternoons when I have time to
              > work on it. Soon after that I'll make the first alpha release.
              >
              > I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
              > Surefire have some overlap, for example both of them take care of
              > classloading and starting a JVM process. One option is to use Surefire's
              > classloading and use only the core runner parts of Jumi. Otherwise Jumi
              > would need information that what are the dependency JARs, what are the
              > current project's classes, maybe also that which of them are test and
              > production classes (Jumi will do caching of the dependencies' class
              > loaders to speed up the tests). That would require changes to Surefire,
              > or then a completely new Maven plugin.
              >
              > kristian@... wrote on 4.1.2012 8:51:
              >
              >
              > > Sorry, I was doing just what I sometimes dislike of our own users ;)
              > >
              > > The real problem I want to solve is associating console output from a
              > > running thread to a given test-case (for parallel runs). We have this
              > > fairly well covered already and we manage to capture everything that
              > > happens within a given test-case for that test-case.
              > >
              > > The real problem is that the RunListener API does not demarcate
              > > test-classes properly, so I am unable to distinguish console output
              > > happening in @AfterClass of one test with @BeforeClass on the test-class
              > > on the same thread.
              > >
              > > This problem has been in the corner of my eye for quite some time and
              > > I've been trying to find a decent angle to solve it from. I promise not
              > > to do anything that is frowned upon, which is why I asked ;)
              > >
              > > Kristian
              > >
              >
              > --
              > Esko Luontola
              > www.orfjackal.net
              >
              >
            Your message has been successfully submitted and would be delivered to recipients shortly.