Loading ...
Sorry, an error occurred while loading the content.

Re: Runner for maven

Expand Messages
  • kristian@rosenvold.com
    maven-surefire now also supports concurrent forks, and they don t have any of the interaction problems you describe - as you know. Since surefire is an
    Message 1 of 12 , Jan 5, 2012
    View Source
    • 0 Attachment
      maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.

      Your first suggestion seems to work quite well, using the Computer I just wrap the created runner in another runner that attaches the testclass to the thread (and delegates on to the target Runner), which we pick up in reporting.

      Surefire does all the other stuff already, but I'm thinking we might
      be deprecating "both" mode threading support because it becomes
      such a pain to do correctly, at least it requires a tight collaboration between the RunListener and the Computer.


      Kristian

      --- In junit@yahoogroups.com, Dawid Weiss <dawid.weiss@...> wrote:
      >
      > Hi Kristian,
      >
      > We had a similar issue. I was debating for a longer while if it makes sense
      > to run parallel tests within the same VM. In many situations this speeds
      > things a lot and has a nice memory conserving footprint but you cannot
      > guarantee tests won't interact with each other (in the simplest case via
      > mutating system properties). Finally the decision was to do it like Apache
      > Lucene did it -- split tests into parallel slave VMs with repeatable suite
      > execution order and sequential test execution within a single VM. This is
      > implemented in an ANT task called junit4 here.
      >
      > https://github.com/carrotsearch/randomizedtesting/tree/master/ant-junit4
      >
      > JUnit tests inside will give you a hint how to use it, the code is still
      > very fresh so documentation is scarce.
      >
      > Now... this was a digression. Back to your question (because we did go that
      > route as well, only backed up), one possible solution to isolating
      > sysout/syserrs of parallel tests in a single VM is:
      >
      > 1) isolate each suite (for @BeforeClass/ @AfterClass hooks) into a separate
      > thread which you have under control. This can be either a custom @Runner or
      > a Computer, many ways to achieve it.
      >
      > 2) write a PrintStream implementation that redirects to a thread-local
      > buffer. This gist implements a similar idea to demux multiple thread
      > streams so that each one's output appears on a single line; you'll get the
      > idea I'm sure.
      >
      > https://gist.github.com/1561833
      >
      > 3) once you have the above the rest is easy: replace default
      > System.out/System.err (I assume no security manager is present or it allows
      > such replacement), clear the buffer before suite, run your suite, flush the
      > buffer. The buffer's content will be your thread's output even though
      > system.* are singletons.
      >
      > Like I said though -- I would consider using multiple jvms for running
      > parallel tests instead.
      >
      > Dawid
      >
      > On Wed, Jan 4, 2012 at 7:51 AM, <kristian@...> wrote:
      >
      > > **
      > >
      > >
      > > Sorry, I was doing just what I sometimes dislike of our own users ;)
      > >
      > > The real problem I want to solve is associating console output from a
      > > running thread to a given test-case (for parallel runs). We have this
      > > fairly well covered already and we manage to capture everything that
      > > happens within a given test-case for that test-case.
      > >
      > > The real problem is that the RunListener API does not demarcate
      > > test-classes properly, so I am unable to distinguish console output
      > > happening in @AfterClass of one test with @BeforeClass on the test-class on
      > > the same thread.
      > >
      > > This problem has been in the corner of my eye for quite some time and I've
      > > been trying to find a decent angle to solve it from. I promise not to do
      > > anything that is frowned upon, which is why I asked ;)
      > >
      > > Kristian
      > >
      > >
      > > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
      > > >
      > > > Kristian,
      > > >
      > > > In reading your original mail, I was also confused as to what you wanted
      > > to
      > > > achieve. Could you expand a little please?
      > > >
      > > > Thanks.
      > > >
      > > > Matthew Farwell.
      > > >
      > > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a écrit :
      > >
      > > >
      > > > > **
      > > > >
      > > > > Hi Kristian
      > > > >
      > > > > Do you really want a transition from RunListener?
      > > > >
      > > > > To an outside observer - a JUnit Runner "describes" its test-executions
      > > > > and reports their outcome to the RunNotifier, which forward the info to
      > > > > RunListener, which is the interface that 3rd-party test-running
      > > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
      > > > >
      > > > > I have seen IDE-project test-extensions try to go beyond
      > > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
      > > > > test-instance creation/setup, method-executions etc) and the result is
      > > > > something I don't like much. It works for simple cases but with
      > > customized
      > > > > Runner-classes and new JUnit-features (e.g. the introduction of
      > > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
      > > > > (i.e. the developers) from taking full advantage of JUnit!
      > > > >
      > > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled
      > > the
      > > > > JUnit-integration wonderfully, as far as I know, so your post worries
      > > me a
      > > > > little. What is it that you want to accomplish by moving away from the
      > > > > successful(!) "Junitcore+RunListener based approach"?
      > > > >
      > > > > Henrik
      > > > >
      > > > >
      > > > > --- In junit@yahoogroups.com, kristian@ wrote:
      > > > > >
      > > > > > Hi,
      > > > > >
      > > > > > I maintain the junit provider for maven-surefire, and I'm looking
      > > into
      > > > > transitioning from a Junitcore+RunListener based approach to an
      > > approach
      > > > > based on something like BlockJUnit4ClassRunner.
      > > > > >
      > > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
      > > > > client code itself may need to do that, so I'd somehow need to put "my"
      > > > > > Runner in front of the client-supplied one, maybe something proxy
      > > like.
      > > > > >
      > > > > > Anyone have any tips on this, maybe some existing code that does
      > > this ?
      > > > > >
      > > > > > Kristian
      > > > > >
      > > > >
      > > > >
      > > > >
      > > >
      > > >
      > > > [Non-text portions of this message have been removed]
      > > >
      > >
      > >
      > >
      >
      >
      > [Non-text portions of this message have been removed]
      >
    • kristian@rosenvold.com
      Surefire is maven s default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I d do in my own tests. If the user decides
      Message 2 of 12 , Jan 5, 2012
      View Source
      • 0 Attachment
        Surefire is maven's default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I'd do in my own tests. If the user decides to use loggers or system out is not really something we want to dictate.

        As a humoristic example, let me mention the issue we had on this guy spawning threads from a test that were producing console output for the entire test run that was collected by a JVM shutdown hook.

        Kristian


        --- In junit@yahoogroups.com, "Georg" <le_garcon_enerve@...> wrote:
        >
        > Dear Kristian,
        >
        > My first thought - and I hope not to sound arrogant - would be to
        > - to remove the the output altogether (who reads it anyhow once the test is run successfully the first time ;-); if one needs to read the output of the test runs, what's the point of unit tests?!).
        > - or to convert it into assertions (they also work nicely in methods not marked with @Test).
        >
        > If you still think you need the output, why not use a logger (log4j), which adds all the information you need? Refactoring should be very easy.
        >
        > As I'm starting to learn and play with AspectJ, I'd try that (but would not bet money on this being a good idea). This would allow to intercept calls to System.out.print() and alike, and tweak it print the argument(s) of the print()-statement augmented with the class to which the calling method belongs.
        >
        >
        >
        >
        > --- In junit@yahoogroups.com, kristian@ wrote:
        > >
        > > Sorry, I was doing just what I sometimes dislike of our own users ;)
        > >
        > > The real problem I want to solve is associating console output from a running thread to a given test-case (for parallel runs). We have this fairly well covered already and we manage to capture everything that happens within a given test-case for that test-case.
        > >
        > > The real problem is that the RunListener API does not demarcate test-classes properly, so I am unable to distinguish console output happening in @AfterClass of one test with @BeforeClass on the test-class on the same thread.
        > >
        > > This problem has been in the corner of my eye for quite some time and I've been trying to find a decent angle to solve it from. I promise not to do anything that is frowned upon, which is why I asked ;)
        > >
        > > Kristian
        > >
        > >
        > > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
        > > >
        > > > Kristian,
        > > >
        > > > In reading your original mail, I was also confused as to what you wanted to
        > > > achieve. Could you expand a little please?
        > > >
        > > > Thanks.
        > > >
        > > > Matthew Farwell.
        > > >
        > > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a écrit :
        > > >
        > > > > **
        > > > >
        > > > > Hi Kristian
        > > > >
        > > > > Do you really want a transition from RunListener?
        > > > >
        > > > > To an outside observer - a JUnit Runner "describes" its test-executions
        > > > > and reports their outcome to the RunNotifier, which forward the info to
        > > > > RunListener, which is the interface that 3rd-party test-running
        > > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
        > > > >
        > > > > I have seen IDE-project test-extensions try to go beyond
        > > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
        > > > > test-instance creation/setup, method-executions etc) and the result is
        > > > > something I don't like much. It works for simple cases but with customized
        > > > > Runner-classes and new JUnit-features (e.g. the introduction of
        > > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
        > > > > (i.e. the developers) from taking full advantage of JUnit!
        > > > >
        > > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
        > > > > JUnit-integration wonderfully, as far as I know, so your post worries me a
        > > > > little. What is it that you want to accomplish by moving away from the
        > > > > successful(!) "Junitcore+RunListener based approach"?
        > > > >
        > > > > Henrik
        > > > >
        > > > >
        > > > > --- In junit@yahoogroups.com, kristian@ wrote:
        > > > > >
        > > > > > Hi,
        > > > > >
        > > > > > I maintain the junit provider for maven-surefire, and I'm looking into
        > > > > transitioning from a Junitcore+RunListener based approach to an approach
        > > > > based on something like BlockJUnit4ClassRunner.
        > > > > >
        > > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
        > > > > client code itself may need to do that, so I'd somehow need to put "my"
        > > > > > Runner in front of the client-supplied one, maybe something proxy like.
        > > > > >
        > > > > > Anyone have any tips on this, maybe some existing code that does this ?
        > > > > >
        > > > > > Kristian
        > > > > >
        > > > >
        > > > >
        > > > >
        > > >
        > > >
        > > > [Non-text portions of this message have been removed]
        > > >
        > >
        >
      • Dawid Weiss
        ... I was thinking about ANT actually, not maven (surefire). I admit we also needed some extra control over suite allocation scheduling and more detailed
        Message 3 of 12 , Jan 5, 2012
        View Source
        • 0 Attachment
          > maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.

          I was thinking about ANT actually, not maven (surefire). I admit we
          also needed some extra control over suite allocation scheduling and
          more detailed reporting, so surefire wasn't a perfect fit for us. It
          is a great tool though, I agree with that.

          Dawid
        • Dawid Weiss
          Oh, I also didn t mention one more thing -- if you have tests that fork subthreads which in turn write to System.out/err then you will also need a smarter way
          Message 4 of 12 , Jan 5, 2012
          View Source
          • 0 Attachment
            Oh, I also didn't mention one more thing -- if you have tests that
            fork subthreads which in turn write to System.out/err then you will
            also need a smarter way to track which suite a thread belongs to.
            This can be done with a parent thread group (in most cases works) but
            like you said -- I think it's just a pain and there is no way to
            implement it correctly for all possible use cases.

            Dawid

            On Thu, Jan 5, 2012 at 12:52 PM, Dawid Weiss <dawid.weiss@...> wrote:
            >> maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.
            >
            > I was thinking about ANT actually, not maven (surefire). I admit we
            > also needed some extra control over suite allocation scheduling and
            > more detailed reporting, so surefire wasn't a perfect fit for us. It
            > is a great tool though, I agree with that.
            >
            > Dawid
          • Esko Luontola
            I m already working on something like that. I m writing a new test runner called Jumi http://jumi.fi/ which will have native support for running tests in
            Message 5 of 12 , Jan 6, 2012
            View Source
            • 0 Attachment
              I'm already working on something like that. I'm writing a new test
              runner called Jumi http://jumi.fi/ which will have native support for
              running tests in parallel in the same JVM, and it take care of capturing
              the System.out/err of individual tests.

              Jumi will do the output capturing by replacing System.out/err with its
              own implementation, which saves all print events (the same way it saves
              test lifecycle events), synchronizes both out and err (I think all
              current IDEs and build tools fail this:
              https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
              and also takes care of capturing the output of threads spawned by the
              test thread. For long-lived threads, such as AWT event dispatch thread,
              I will also try to figure out something (I'll timestamp all print
              events, so it should be possible to figure out roughly which test
              printed things in the AWT thread).

              The output capturing is at the top of my TODO list, so I will have it
              implemented after a couple of productive afternoons when I have time to
              work on it. Soon after that I'll make the first alpha release.

              I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
              Surefire have some overlap, for example both of them take care of
              classloading and starting a JVM process. One option is to use Surefire's
              classloading and use only the core runner parts of Jumi. Otherwise Jumi
              would need information that what are the dependency JARs, what are the
              current project's classes, maybe also that which of them are test and
              production classes (Jumi will do caching of the dependencies' class
              loaders to speed up the tests). That would require changes to Surefire,
              or then a completely new Maven plugin.


              kristian@... wrote on 4.1.2012 8:51:
              > Sorry, I was doing just what I sometimes dislike of our own users ;)
              >
              > The real problem I want to solve is associating console output from a
              > running thread to a given test-case (for parallel runs). We have this
              > fairly well covered already and we manage to capture everything that
              > happens within a given test-case for that test-case.
              >
              > The real problem is that the RunListener API does not demarcate
              > test-classes properly, so I am unable to distinguish console output
              > happening in @AfterClass of one test with @BeforeClass on the test-class
              > on the same thread.
              >
              > This problem has been in the corner of my eye for quite some time and
              > I've been trying to find a decent angle to solve it from. I promise not
              > to do anything that is frowned upon, which is why I asked ;)
              >
              > Kristian
              >

              --
              Esko Luontola
              www.orfjackal.net
            • Dawid Weiss
              You may want to look into RandomizedRunner/JUnit4 task -- it has most of the things you mentioned (synchronized i/o events with reporting, capturing runaway
              Message 6 of 12 , Jan 6, 2012
              View Source
              • 0 Attachment
                You may want to look into RandomizedRunner/JUnit4 task -- it has most
                of the things you mentioned (synchronized i/o events with reporting,
                capturing runaway threads and the like).

                The code is here:
                https://github.com/carrotsearch/randomizedtesting/

                I have a talk about randomized testing which includes the above
                concepts at Lucene EuroCon Barcelona, the video is here:
                http://vimeo.com/32087114

                Dawid

                On Fri, Jan 6, 2012 at 1:29 PM, Esko Luontola <esko.luontola@...> wrote:
                >
                >
                >
                > I'm already working on something like that. I'm writing a new test
                > runner called Jumi http://jumi.fi/ which will have native support for
                > running tests in parallel in the same JVM, and it take care of capturing
                > the System.out/err of individual tests.
                >
                > Jumi will do the output capturing by replacing System.out/err with its
                > own implementation, which saves all print events (the same way it saves
                > test lifecycle events), synchronizes both out and err (I think all
                > current IDEs and build tools fail this:
                > https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
                > and also takes care of capturing the output of threads spawned by the
                > test thread. For long-lived threads, such as AWT event dispatch thread,
                > I will also try to figure out something (I'll timestamp all print
                > events, so it should be possible to figure out roughly which test
                > printed things in the AWT thread).
                >
                > The output capturing is at the top of my TODO list, so I will have it
                > implemented after a couple of productive afternoons when I have time to
                > work on it. Soon after that I'll make the first alpha release.
                >
                > I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
                > Surefire have some overlap, for example both of them take care of
                > classloading and starting a JVM process. One option is to use Surefire's
                > classloading and use only the core runner parts of Jumi. Otherwise Jumi
                > would need information that what are the dependency JARs, what are the
                > current project's classes, maybe also that which of them are test and
                > production classes (Jumi will do caching of the dependencies' class
                > loaders to speed up the tests). That would require changes to Surefire,
                > or then a completely new Maven plugin.
                >
                > kristian@... wrote on 4.1.2012 8:51:
                >
                >
                > > Sorry, I was doing just what I sometimes dislike of our own users ;)
                > >
                > > The real problem I want to solve is associating console output from a
                > > running thread to a given test-case (for parallel runs). We have this
                > > fairly well covered already and we manage to capture everything that
                > > happens within a given test-case for that test-case.
                > >
                > > The real problem is that the RunListener API does not demarcate
                > > test-classes properly, so I am unable to distinguish console output
                > > happening in @AfterClass of one test with @BeforeClass on the test-class
                > > on the same thread.
                > >
                > > This problem has been in the corner of my eye for quite some time and
                > > I've been trying to find a decent angle to solve it from. I promise not
                > > to do anything that is frowned upon, which is why I asked ;)
                > >
                > > Kristian
                > >
                >
                > --
                > Esko Luontola
                > www.orfjackal.net
                >
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.