Loading ...
Sorry, an error occurred while loading the content.

Runner for maven

Expand Messages
  • kristian@rosenvold.com
    Hi, I maintain the junit provider for maven-surefire, and I m looking into transitioning from a Junitcore+RunListener based approach to an approach based on
    Message 1 of 12 , Jan 2, 2012
    View Source
    • 0 Attachment
      Hi,

      I maintain the junit provider for maven-surefire, and I'm looking into transitioning from a Junitcore+RunListener based approach to an approach based on something like BlockJUnit4ClassRunner.

      Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the client code itself may need to do that, so I'd somehow need to put "my"
      Runner in front of the client-supplied one, maybe something proxy like.

      Anyone have any tips on this, maybe some existing code that does this ?

      Kristian
    • hkaipe
      Hi Kristian Do you really want a transition from RunListener? To an outside observer - a JUnit Runner describes its test-executions and reports their outcome
      Message 2 of 12 , Jan 3, 2012
      View Source
      • 0 Attachment
        Hi Kristian

        Do you really want a transition from RunListener?

        To an outside observer - a JUnit Runner "describes" its test-executions and reports their outcome to the RunNotifier, which forward the info to RunListener, which is the interface that 3rd-party test-running applications (e.g. IDEs, Maven-Surefire etc) are expected to use.

        I have seen IDE-project test-extensions try to go beyond RunListener/RunNotifier and start to joggle with Runner internals (e.g. test-instance creation/setup, method-executions etc) and the result is something I don't like much. It works for simple cases but with customized Runner-classes and new JUnit-features (e.g. the introduction of JUnit-Rules) the result is usually bad and it discourages the IDE users (i.e. the developers) from taking full advantage of JUnit!

        However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the JUnit-integration wonderfully, as far as I know, so your post worries me a little. What is it that you want to accomplish by moving away from the successful(!) "Junitcore+RunListener based approach"?

        Henrik


        --- In junit@yahoogroups.com, kristian@... wrote:
        >
        > Hi,
        >
        > I maintain the junit provider for maven-surefire, and I'm looking into transitioning from a Junitcore+RunListener based approach to an approach based on something like BlockJUnit4ClassRunner.
        >
        > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the client code itself may need to do that, so I'd somehow need to put "my"
        > Runner in front of the client-supplied one, maybe something proxy like.
        >
        > Anyone have any tips on this, maybe some existing code that does this ?
        >
        > Kristian
        >
      • Matthew Farwell
        Kristian, In reading your original mail, I was also confused as to what you wanted to achieve. Could you expand a little please? Thanks. Matthew Farwell. ...
        Message 3 of 12 , Jan 3, 2012
        View Source
        • 0 Attachment
          Kristian,

          In reading your original mail, I was also confused as to what you wanted to
          achieve. Could you expand a little please?

          Thanks.

          Matthew Farwell.

          Le 3 janvier 2012 22:45, hkaipe <hkaipe@...> a �crit :

          > **
          >
          > Hi Kristian
          >
          > Do you really want a transition from RunListener?
          >
          > To an outside observer - a JUnit Runner "describes" its test-executions
          > and reports their outcome to the RunNotifier, which forward the info to
          > RunListener, which is the interface that 3rd-party test-running
          > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
          >
          > I have seen IDE-project test-extensions try to go beyond
          > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
          > test-instance creation/setup, method-executions etc) and the result is
          > something I don't like much. It works for simple cases but with customized
          > Runner-classes and new JUnit-features (e.g. the introduction of
          > JUnit-Rules) the result is usually bad and it discourages the IDE users
          > (i.e. the developers) from taking full advantage of JUnit!
          >
          > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
          > JUnit-integration wonderfully, as far as I know, so your post worries me a
          > little. What is it that you want to accomplish by moving away from the
          > successful(!) "Junitcore+RunListener based approach"?
          >
          > Henrik
          >
          >
          > --- In junit@yahoogroups.com, kristian@... wrote:
          > >
          > > Hi,
          > >
          > > I maintain the junit provider for maven-surefire, and I'm looking into
          > transitioning from a Junitcore+RunListener based approach to an approach
          > based on something like BlockJUnit4ClassRunner.
          > >
          > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
          > client code itself may need to do that, so I'd somehow need to put "my"
          > > Runner in front of the client-supplied one, maybe something proxy like.
          > >
          > > Anyone have any tips on this, maybe some existing code that does this ?
          > >
          > > Kristian
          > >
          >
          >
          >


          [Non-text portions of this message have been removed]
        • kristian@rosenvold.com
          Sorry, I was doing just what I sometimes dislike of our own users ;) The real problem I want to solve is associating console output from a running thread to a
          Message 4 of 12 , Jan 3, 2012
          View Source
          • 0 Attachment
            Sorry, I was doing just what I sometimes dislike of our own users ;)

            The real problem I want to solve is associating console output from a running thread to a given test-case (for parallel runs). We have this fairly well covered already and we manage to capture everything that happens within a given test-case for that test-case.

            The real problem is that the RunListener API does not demarcate test-classes properly, so I am unable to distinguish console output happening in @AfterClass of one test with @BeforeClass on the test-class on the same thread.

            This problem has been in the corner of my eye for quite some time and I've been trying to find a decent angle to solve it from. I promise not to do anything that is frowned upon, which is why I asked ;)

            Kristian


            --- In junit@yahoogroups.com, Matthew Farwell <matthew@...> wrote:
            >
            > Kristian,
            >
            > In reading your original mail, I was also confused as to what you wanted to
            > achieve. Could you expand a little please?
            >
            > Thanks.
            >
            > Matthew Farwell.
            >
            > Le 3 janvier 2012 22:45, hkaipe <hkaipe@...> a écrit :
            >
            > > **
            > >
            > > Hi Kristian
            > >
            > > Do you really want a transition from RunListener?
            > >
            > > To an outside observer - a JUnit Runner "describes" its test-executions
            > > and reports their outcome to the RunNotifier, which forward the info to
            > > RunListener, which is the interface that 3rd-party test-running
            > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
            > >
            > > I have seen IDE-project test-extensions try to go beyond
            > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
            > > test-instance creation/setup, method-executions etc) and the result is
            > > something I don't like much. It works for simple cases but with customized
            > > Runner-classes and new JUnit-features (e.g. the introduction of
            > > JUnit-Rules) the result is usually bad and it discourages the IDE users
            > > (i.e. the developers) from taking full advantage of JUnit!
            > >
            > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
            > > JUnit-integration wonderfully, as far as I know, so your post worries me a
            > > little. What is it that you want to accomplish by moving away from the
            > > successful(!) "Junitcore+RunListener based approach"?
            > >
            > > Henrik
            > >
            > >
            > > --- In junit@yahoogroups.com, kristian@ wrote:
            > > >
            > > > Hi,
            > > >
            > > > I maintain the junit provider for maven-surefire, and I'm looking into
            > > transitioning from a Junitcore+RunListener based approach to an approach
            > > based on something like BlockJUnit4ClassRunner.
            > > >
            > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
            > > client code itself may need to do that, so I'd somehow need to put "my"
            > > > Runner in front of the client-supplied one, maybe something proxy like.
            > > >
            > > > Anyone have any tips on this, maybe some existing code that does this ?
            > > >
            > > > Kristian
            > > >
            > >
            > >
            > >
            >
            >
            > [Non-text portions of this message have been removed]
            >
          • Georg
            Dear Kristian, My first thought - and I hope not to sound arrogant - would be to - to remove the the output altogether (who reads it anyhow once the test is
            Message 5 of 12 , Jan 4, 2012
            View Source
            • 0 Attachment
              Dear Kristian,

              My first thought - and I hope not to sound arrogant - would be to
              - to remove the the output altogether (who reads it anyhow once the test is run successfully the first time ;-); if one needs to read the output of the test runs, what's the point of unit tests?!).
              - or to convert it into assertions (they also work nicely in methods not marked with @Test).

              If you still think you need the output, why not use a logger (log4j), which adds all the information you need? Refactoring should be very easy.

              As I'm starting to learn and play with AspectJ, I'd try that (but would not bet money on this being a good idea). This would allow to intercept calls to System.out.print() and alike, and tweak it print the argument(s) of the print()-statement augmented with the class to which the calling method belongs.




              --- In junit@yahoogroups.com, kristian@... wrote:
              >
              > Sorry, I was doing just what I sometimes dislike of our own users ;)
              >
              > The real problem I want to solve is associating console output from a running thread to a given test-case (for parallel runs). We have this fairly well covered already and we manage to capture everything that happens within a given test-case for that test-case.
              >
              > The real problem is that the RunListener API does not demarcate test-classes properly, so I am unable to distinguish console output happening in @AfterClass of one test with @BeforeClass on the test-class on the same thread.
              >
              > This problem has been in the corner of my eye for quite some time and I've been trying to find a decent angle to solve it from. I promise not to do anything that is frowned upon, which is why I asked ;)
              >
              > Kristian
              >
              >
              > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
              > >
              > > Kristian,
              > >
              > > In reading your original mail, I was also confused as to what you wanted to
              > > achieve. Could you expand a little please?
              > >
              > > Thanks.
              > >
              > > Matthew Farwell.
              > >
              > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a Иcrit :
              > >
              > > > **
              > > >
              > > > Hi Kristian
              > > >
              > > > Do you really want a transition from RunListener?
              > > >
              > > > To an outside observer - a JUnit Runner "describes" its test-executions
              > > > and reports their outcome to the RunNotifier, which forward the info to
              > > > RunListener, which is the interface that 3rd-party test-running
              > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
              > > >
              > > > I have seen IDE-project test-extensions try to go beyond
              > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
              > > > test-instance creation/setup, method-executions etc) and the result is
              > > > something I don't like much. It works for simple cases but with customized
              > > > Runner-classes and new JUnit-features (e.g. the introduction of
              > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
              > > > (i.e. the developers) from taking full advantage of JUnit!
              > > >
              > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
              > > > JUnit-integration wonderfully, as far as I know, so your post worries me a
              > > > little. What is it that you want to accomplish by moving away from the
              > > > successful(!) "Junitcore+RunListener based approach"?
              > > >
              > > > Henrik
              > > >
              > > >
              > > > --- In junit@yahoogroups.com, kristian@ wrote:
              > > > >
              > > > > Hi,
              > > > >
              > > > > I maintain the junit provider for maven-surefire, and I'm looking into
              > > > transitioning from a Junitcore+RunListener based approach to an approach
              > > > based on something like BlockJUnit4ClassRunner.
              > > > >
              > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
              > > > client code itself may need to do that, so I'd somehow need to put "my"
              > > > > Runner in front of the client-supplied one, maybe something proxy like.
              > > > >
              > > > > Anyone have any tips on this, maybe some existing code that does this ?
              > > > >
              > > > > Kristian
              > > > >
              > > >
              > > >
              > > >
              > >
              > >
              > > [Non-text portions of this message have been removed]
              > >
              >
            • Dawid Weiss
              Hi Kristian, We had a similar issue. I was debating for a longer while if it makes sense to run parallel tests within the same VM. In many situations this
              Message 6 of 12 , Jan 4, 2012
              View Source
              • 0 Attachment
                Hi Kristian,

                We had a similar issue. I was debating for a longer while if it makes sense
                to run parallel tests within the same VM. In many situations this speeds
                things a lot and has a nice memory conserving footprint but you cannot
                guarantee tests won't interact with each other (in the simplest case via
                mutating system properties). Finally the decision was to do it like Apache
                Lucene did it -- split tests into parallel slave VMs with repeatable suite
                execution order and sequential test execution within a single VM. This is
                implemented in an ANT task called junit4 here.

                https://github.com/carrotsearch/randomizedtesting/tree/master/ant-junit4

                JUnit tests inside will give you a hint how to use it, the code is still
                very fresh so documentation is scarce.

                Now... this was a digression. Back to your question (because we did go that
                route as well, only backed up), one possible solution to isolating
                sysout/syserrs of parallel tests in a single VM is:

                1) isolate each suite (for @BeforeClass/ @AfterClass hooks) into a separate
                thread which you have under control. This can be either a custom @Runner or
                a Computer, many ways to achieve it.

                2) write a PrintStream implementation that redirects to a thread-local
                buffer. This gist implements a similar idea to demux multiple thread
                streams so that each one's output appears on a single line; you'll get the
                idea I'm sure.

                https://gist.github.com/1561833

                3) once you have the above the rest is easy: replace default
                System.out/System.err (I assume no security manager is present or it allows
                such replacement), clear the buffer before suite, run your suite, flush the
                buffer. The buffer's content will be your thread's output even though
                system.* are singletons.

                Like I said though -- I would consider using multiple jvms for running
                parallel tests instead.

                Dawid

                On Wed, Jan 4, 2012 at 7:51 AM, <kristian@...> wrote:

                > **
                >
                >
                > Sorry, I was doing just what I sometimes dislike of our own users ;)
                >
                > The real problem I want to solve is associating console output from a
                > running thread to a given test-case (for parallel runs). We have this
                > fairly well covered already and we manage to capture everything that
                > happens within a given test-case for that test-case.
                >
                > The real problem is that the RunListener API does not demarcate
                > test-classes properly, so I am unable to distinguish console output
                > happening in @AfterClass of one test with @BeforeClass on the test-class on
                > the same thread.
                >
                > This problem has been in the corner of my eye for quite some time and I've
                > been trying to find a decent angle to solve it from. I promise not to do
                > anything that is frowned upon, which is why I asked ;)
                >
                > Kristian
                >
                >
                > --- In junit@yahoogroups.com, Matthew Farwell <matthew@...> wrote:
                > >
                > > Kristian,
                > >
                > > In reading your original mail, I was also confused as to what you wanted
                > to
                > > achieve. Could you expand a little please?
                > >
                > > Thanks.
                > >
                > > Matthew Farwell.
                > >
                > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@...> a écrit :
                >
                > >
                > > > **
                > > >
                > > > Hi Kristian
                > > >
                > > > Do you really want a transition from RunListener?
                > > >
                > > > To an outside observer - a JUnit Runner "describes" its test-executions
                > > > and reports their outcome to the RunNotifier, which forward the info to
                > > > RunListener, which is the interface that 3rd-party test-running
                > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
                > > >
                > > > I have seen IDE-project test-extensions try to go beyond
                > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
                > > > test-instance creation/setup, method-executions etc) and the result is
                > > > something I don't like much. It works for simple cases but with
                > customized
                > > > Runner-classes and new JUnit-features (e.g. the introduction of
                > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
                > > > (i.e. the developers) from taking full advantage of JUnit!
                > > >
                > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled
                > the
                > > > JUnit-integration wonderfully, as far as I know, so your post worries
                > me a
                > > > little. What is it that you want to accomplish by moving away from the
                > > > successful(!) "Junitcore+RunListener based approach"?
                > > >
                > > > Henrik
                > > >
                > > >
                > > > --- In junit@yahoogroups.com, kristian@ wrote:
                > > > >
                > > > > Hi,
                > > > >
                > > > > I maintain the junit provider for maven-surefire, and I'm looking
                > into
                > > > transitioning from a Junitcore+RunListener based approach to an
                > approach
                > > > based on something like BlockJUnit4ClassRunner.
                > > > >
                > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
                > > > client code itself may need to do that, so I'd somehow need to put "my"
                > > > > Runner in front of the client-supplied one, maybe something proxy
                > like.
                > > > >
                > > > > Anyone have any tips on this, maybe some existing code that does
                > this ?
                > > > >
                > > > > Kristian
                > > > >
                > > >
                > > >
                > > >
                > >
                > >
                > > [Non-text portions of this message have been removed]
                > >
                >
                >
                >


                [Non-text portions of this message have been removed]
              • kristian@rosenvold.com
                maven-surefire now also supports concurrent forks, and they don t have any of the interaction problems you describe - as you know. Since surefire is an
                Message 7 of 12 , Jan 5, 2012
                View Source
                • 0 Attachment
                  maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.

                  Your first suggestion seems to work quite well, using the Computer I just wrap the created runner in another runner that attaches the testclass to the thread (and delegates on to the target Runner), which we pick up in reporting.

                  Surefire does all the other stuff already, but I'm thinking we might
                  be deprecating "both" mode threading support because it becomes
                  such a pain to do correctly, at least it requires a tight collaboration between the RunListener and the Computer.


                  Kristian

                  --- In junit@yahoogroups.com, Dawid Weiss <dawid.weiss@...> wrote:
                  >
                  > Hi Kristian,
                  >
                  > We had a similar issue. I was debating for a longer while if it makes sense
                  > to run parallel tests within the same VM. In many situations this speeds
                  > things a lot and has a nice memory conserving footprint but you cannot
                  > guarantee tests won't interact with each other (in the simplest case via
                  > mutating system properties). Finally the decision was to do it like Apache
                  > Lucene did it -- split tests into parallel slave VMs with repeatable suite
                  > execution order and sequential test execution within a single VM. This is
                  > implemented in an ANT task called junit4 here.
                  >
                  > https://github.com/carrotsearch/randomizedtesting/tree/master/ant-junit4
                  >
                  > JUnit tests inside will give you a hint how to use it, the code is still
                  > very fresh so documentation is scarce.
                  >
                  > Now... this was a digression. Back to your question (because we did go that
                  > route as well, only backed up), one possible solution to isolating
                  > sysout/syserrs of parallel tests in a single VM is:
                  >
                  > 1) isolate each suite (for @BeforeClass/ @AfterClass hooks) into a separate
                  > thread which you have under control. This can be either a custom @Runner or
                  > a Computer, many ways to achieve it.
                  >
                  > 2) write a PrintStream implementation that redirects to a thread-local
                  > buffer. This gist implements a similar idea to demux multiple thread
                  > streams so that each one's output appears on a single line; you'll get the
                  > idea I'm sure.
                  >
                  > https://gist.github.com/1561833
                  >
                  > 3) once you have the above the rest is easy: replace default
                  > System.out/System.err (I assume no security manager is present or it allows
                  > such replacement), clear the buffer before suite, run your suite, flush the
                  > buffer. The buffer's content will be your thread's output even though
                  > system.* are singletons.
                  >
                  > Like I said though -- I would consider using multiple jvms for running
                  > parallel tests instead.
                  >
                  > Dawid
                  >
                  > On Wed, Jan 4, 2012 at 7:51 AM, <kristian@...> wrote:
                  >
                  > > **
                  > >
                  > >
                  > > Sorry, I was doing just what I sometimes dislike of our own users ;)
                  > >
                  > > The real problem I want to solve is associating console output from a
                  > > running thread to a given test-case (for parallel runs). We have this
                  > > fairly well covered already and we manage to capture everything that
                  > > happens within a given test-case for that test-case.
                  > >
                  > > The real problem is that the RunListener API does not demarcate
                  > > test-classes properly, so I am unable to distinguish console output
                  > > happening in @AfterClass of one test with @BeforeClass on the test-class on
                  > > the same thread.
                  > >
                  > > This problem has been in the corner of my eye for quite some time and I've
                  > > been trying to find a decent angle to solve it from. I promise not to do
                  > > anything that is frowned upon, which is why I asked ;)
                  > >
                  > > Kristian
                  > >
                  > >
                  > > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
                  > > >
                  > > > Kristian,
                  > > >
                  > > > In reading your original mail, I was also confused as to what you wanted
                  > > to
                  > > > achieve. Could you expand a little please?
                  > > >
                  > > > Thanks.
                  > > >
                  > > > Matthew Farwell.
                  > > >
                  > > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a écrit :
                  > >
                  > > >
                  > > > > **
                  > > > >
                  > > > > Hi Kristian
                  > > > >
                  > > > > Do you really want a transition from RunListener?
                  > > > >
                  > > > > To an outside observer - a JUnit Runner "describes" its test-executions
                  > > > > and reports their outcome to the RunNotifier, which forward the info to
                  > > > > RunListener, which is the interface that 3rd-party test-running
                  > > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
                  > > > >
                  > > > > I have seen IDE-project test-extensions try to go beyond
                  > > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
                  > > > > test-instance creation/setup, method-executions etc) and the result is
                  > > > > something I don't like much. It works for simple cases but with
                  > > customized
                  > > > > Runner-classes and new JUnit-features (e.g. the introduction of
                  > > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
                  > > > > (i.e. the developers) from taking full advantage of JUnit!
                  > > > >
                  > > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled
                  > > the
                  > > > > JUnit-integration wonderfully, as far as I know, so your post worries
                  > > me a
                  > > > > little. What is it that you want to accomplish by moving away from the
                  > > > > successful(!) "Junitcore+RunListener based approach"?
                  > > > >
                  > > > > Henrik
                  > > > >
                  > > > >
                  > > > > --- In junit@yahoogroups.com, kristian@ wrote:
                  > > > > >
                  > > > > > Hi,
                  > > > > >
                  > > > > > I maintain the junit provider for maven-surefire, and I'm looking
                  > > into
                  > > > > transitioning from a Junitcore+RunListener based approach to an
                  > > approach
                  > > > > based on something like BlockJUnit4ClassRunner.
                  > > > > >
                  > > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
                  > > > > client code itself may need to do that, so I'd somehow need to put "my"
                  > > > > > Runner in front of the client-supplied one, maybe something proxy
                  > > like.
                  > > > > >
                  > > > > > Anyone have any tips on this, maybe some existing code that does
                  > > this ?
                  > > > > >
                  > > > > > Kristian
                  > > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > >
                  > > >
                  > > > [Non-text portions of this message have been removed]
                  > > >
                  > >
                  > >
                  > >
                  >
                  >
                  > [Non-text portions of this message have been removed]
                  >
                • kristian@rosenvold.com
                  Surefire is maven s default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I d do in my own tests. If the user decides
                  Message 8 of 12 , Jan 5, 2012
                  View Source
                  • 0 Attachment
                    Surefire is maven's default test-runner. It serves a bucketload of use-cases, some of which are *far* off from what I'd do in my own tests. If the user decides to use loggers or system out is not really something we want to dictate.

                    As a humoristic example, let me mention the issue we had on this guy spawning threads from a test that were producing console output for the entire test run that was collected by a JVM shutdown hook.

                    Kristian


                    --- In junit@yahoogroups.com, "Georg" <le_garcon_enerve@...> wrote:
                    >
                    > Dear Kristian,
                    >
                    > My first thought - and I hope not to sound arrogant - would be to
                    > - to remove the the output altogether (who reads it anyhow once the test is run successfully the first time ;-); if one needs to read the output of the test runs, what's the point of unit tests?!).
                    > - or to convert it into assertions (they also work nicely in methods not marked with @Test).
                    >
                    > If you still think you need the output, why not use a logger (log4j), which adds all the information you need? Refactoring should be very easy.
                    >
                    > As I'm starting to learn and play with AspectJ, I'd try that (but would not bet money on this being a good idea). This would allow to intercept calls to System.out.print() and alike, and tweak it print the argument(s) of the print()-statement augmented with the class to which the calling method belongs.
                    >
                    >
                    >
                    >
                    > --- In junit@yahoogroups.com, kristian@ wrote:
                    > >
                    > > Sorry, I was doing just what I sometimes dislike of our own users ;)
                    > >
                    > > The real problem I want to solve is associating console output from a running thread to a given test-case (for parallel runs). We have this fairly well covered already and we manage to capture everything that happens within a given test-case for that test-case.
                    > >
                    > > The real problem is that the RunListener API does not demarcate test-classes properly, so I am unable to distinguish console output happening in @AfterClass of one test with @BeforeClass on the test-class on the same thread.
                    > >
                    > > This problem has been in the corner of my eye for quite some time and I've been trying to find a decent angle to solve it from. I promise not to do anything that is frowned upon, which is why I asked ;)
                    > >
                    > > Kristian
                    > >
                    > >
                    > > --- In junit@yahoogroups.com, Matthew Farwell <matthew@> wrote:
                    > > >
                    > > > Kristian,
                    > > >
                    > > > In reading your original mail, I was also confused as to what you wanted to
                    > > > achieve. Could you expand a little please?
                    > > >
                    > > > Thanks.
                    > > >
                    > > > Matthew Farwell.
                    > > >
                    > > > Le 3 janvier 2012 22:45, hkaipe <hkaipe@> a écrit :
                    > > >
                    > > > > **
                    > > > >
                    > > > > Hi Kristian
                    > > > >
                    > > > > Do you really want a transition from RunListener?
                    > > > >
                    > > > > To an outside observer - a JUnit Runner "describes" its test-executions
                    > > > > and reports their outcome to the RunNotifier, which forward the info to
                    > > > > RunListener, which is the interface that 3rd-party test-running
                    > > > > applications (e.g. IDEs, Maven-Surefire etc) are expected to use.
                    > > > >
                    > > > > I have seen IDE-project test-extensions try to go beyond
                    > > > > RunListener/RunNotifier and start to joggle with Runner internals (e.g.
                    > > > > test-instance creation/setup, method-executions etc) and the result is
                    > > > > something I don't like much. It works for simple cases but with customized
                    > > > > Runner-classes and new JUnit-features (e.g. the introduction of
                    > > > > JUnit-Rules) the result is usually bad and it discourages the IDE users
                    > > > > (i.e. the developers) from taking full advantage of JUnit!
                    > > > >
                    > > > > However, Maven (versions 2.0-2.2) and its Surefire-plugin has handled the
                    > > > > JUnit-integration wonderfully, as far as I know, so your post worries me a
                    > > > > little. What is it that you want to accomplish by moving away from the
                    > > > > successful(!) "Junitcore+RunListener based approach"?
                    > > > >
                    > > > > Henrik
                    > > > >
                    > > > >
                    > > > > --- In junit@yahoogroups.com, kristian@ wrote:
                    > > > > >
                    > > > > > Hi,
                    > > > > >
                    > > > > > I maintain the junit provider for maven-surefire, and I'm looking into
                    > > > > transitioning from a Junitcore+RunListener based approach to an approach
                    > > > > based on something like BlockJUnit4ClassRunner.
                    > > > > >
                    > > > > > Unfortunately I cannot subclass the BlockJUnit4ClassRunner since the
                    > > > > client code itself may need to do that, so I'd somehow need to put "my"
                    > > > > > Runner in front of the client-supplied one, maybe something proxy like.
                    > > > > >
                    > > > > > Anyone have any tips on this, maybe some existing code that does this ?
                    > > > > >
                    > > > > > Kristian
                    > > > > >
                    > > > >
                    > > > >
                    > > > >
                    > > >
                    > > >
                    > > > [Non-text portions of this message have been removed]
                    > > >
                    > >
                    >
                  • Dawid Weiss
                    ... I was thinking about ANT actually, not maven (surefire). I admit we also needed some extra control over suite allocation scheduling and more detailed
                    Message 9 of 12 , Jan 5, 2012
                    View Source
                    • 0 Attachment
                      > maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.

                      I was thinking about ANT actually, not maven (surefire). I admit we
                      also needed some extra control over suite allocation scheduling and
                      more detailed reporting, so surefire wasn't a perfect fit for us. It
                      is a great tool though, I agree with that.

                      Dawid
                    • Dawid Weiss
                      Oh, I also didn t mention one more thing -- if you have tests that fork subthreads which in turn write to System.out/err then you will also need a smarter way
                      Message 10 of 12 , Jan 5, 2012
                      View Source
                      • 0 Attachment
                        Oh, I also didn't mention one more thing -- if you have tests that
                        fork subthreads which in turn write to System.out/err then you will
                        also need a smarter way to track which suite a thread belongs to.
                        This can be done with a parent thread group (in most cases works) but
                        like you said -- I think it's just a pain and there is no way to
                        implement it correctly for all possible use cases.

                        Dawid

                        On Thu, Jan 5, 2012 at 12:52 PM, Dawid Weiss <dawid.weiss@...> wrote:
                        >> maven-surefire now also supports concurrent forks, and they don't have any of the interaction problems you describe - as you know. Since surefire is an > all-purpose runner, we try to support any "sensible" mode of execution, and there are good use cases for both models of execution.
                        >
                        > I was thinking about ANT actually, not maven (surefire). I admit we
                        > also needed some extra control over suite allocation scheduling and
                        > more detailed reporting, so surefire wasn't a perfect fit for us. It
                        > is a great tool though, I agree with that.
                        >
                        > Dawid
                      • Esko Luontola
                        I m already working on something like that. I m writing a new test runner called Jumi http://jumi.fi/ which will have native support for running tests in
                        Message 11 of 12 , Jan 6, 2012
                        View Source
                        • 0 Attachment
                          I'm already working on something like that. I'm writing a new test
                          runner called Jumi http://jumi.fi/ which will have native support for
                          running tests in parallel in the same JVM, and it take care of capturing
                          the System.out/err of individual tests.

                          Jumi will do the output capturing by replacing System.out/err with its
                          own implementation, which saves all print events (the same way it saves
                          test lifecycle events), synchronizes both out and err (I think all
                          current IDEs and build tools fail this:
                          https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
                          and also takes care of capturing the output of threads spawned by the
                          test thread. For long-lived threads, such as AWT event dispatch thread,
                          I will also try to figure out something (I'll timestamp all print
                          events, so it should be possible to figure out roughly which test
                          printed things in the AWT thread).

                          The output capturing is at the top of my TODO list, so I will have it
                          implemented after a couple of productive afternoons when I have time to
                          work on it. Soon after that I'll make the first alpha release.

                          I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
                          Surefire have some overlap, for example both of them take care of
                          classloading and starting a JVM process. One option is to use Surefire's
                          classloading and use only the core runner parts of Jumi. Otherwise Jumi
                          would need information that what are the dependency JARs, what are the
                          current project's classes, maybe also that which of them are test and
                          production classes (Jumi will do caching of the dependencies' class
                          loaders to speed up the tests). That would require changes to Surefire,
                          or then a completely new Maven plugin.


                          kristian@... wrote on 4.1.2012 8:51:
                          > Sorry, I was doing just what I sometimes dislike of our own users ;)
                          >
                          > The real problem I want to solve is associating console output from a
                          > running thread to a given test-case (for parallel runs). We have this
                          > fairly well covered already and we manage to capture everything that
                          > happens within a given test-case for that test-case.
                          >
                          > The real problem is that the RunListener API does not demarcate
                          > test-classes properly, so I am unable to distinguish console output
                          > happening in @AfterClass of one test with @BeforeClass on the test-class
                          > on the same thread.
                          >
                          > This problem has been in the corner of my eye for quite some time and
                          > I've been trying to find a decent angle to solve it from. I promise not
                          > to do anything that is frowned upon, which is why I asked ;)
                          >
                          > Kristian
                          >

                          --
                          Esko Luontola
                          www.orfjackal.net
                        • Dawid Weiss
                          You may want to look into RandomizedRunner/JUnit4 task -- it has most of the things you mentioned (synchronized i/o events with reporting, capturing runaway
                          Message 12 of 12 , Jan 6, 2012
                          View Source
                          • 0 Attachment
                            You may want to look into RandomizedRunner/JUnit4 task -- it has most
                            of the things you mentioned (synchronized i/o events with reporting,
                            capturing runaway threads and the like).

                            The code is here:
                            https://github.com/carrotsearch/randomizedtesting/

                            I have a talk about randomized testing which includes the above
                            concepts at Lucene EuroCon Barcelona, the video is here:
                            http://vimeo.com/32087114

                            Dawid

                            On Fri, Jan 6, 2012 at 1:29 PM, Esko Luontola <esko.luontola@...> wrote:
                            >
                            >
                            >
                            > I'm already working on something like that. I'm writing a new test
                            > runner called Jumi http://jumi.fi/ which will have native support for
                            > running tests in parallel in the same JVM, and it take care of capturing
                            > the System.out/err of individual tests.
                            >
                            > Jumi will do the output capturing by replacing System.out/err with its
                            > own implementation, which saves all print events (the same way it saves
                            > test lifecycle events), synchronizes both out and err (I think all
                            > current IDEs and build tools fail this:
                            > https://github.com/orfjackal/misc-tools/blob/master/src/test/java/net/orfjackal/bugs/SystemOutErrBugTest.java),
                            > and also takes care of capturing the output of threads spawned by the
                            > test thread. For long-lived threads, such as AWT event dispatch thread,
                            > I will also try to figure out something (I'll timestamp all print
                            > events, so it should be possible to figure out roughly which test
                            > printed things in the AWT thread).
                            >
                            > The output capturing is at the top of my TODO list, so I will have it
                            > implemented after a couple of productive afternoons when I have time to
                            > work on it. Soon after that I'll make the first alpha release.
                            >
                            > I've looked a bit at how to integrate Jumi with maven-surefire. Jumi and
                            > Surefire have some overlap, for example both of them take care of
                            > classloading and starting a JVM process. One option is to use Surefire's
                            > classloading and use only the core runner parts of Jumi. Otherwise Jumi
                            > would need information that what are the dependency JARs, what are the
                            > current project's classes, maybe also that which of them are test and
                            > production classes (Jumi will do caching of the dependencies' class
                            > loaders to speed up the tests). That would require changes to Surefire,
                            > or then a completely new Maven plugin.
                            >
                            > kristian@... wrote on 4.1.2012 8:51:
                            >
                            >
                            > > Sorry, I was doing just what I sometimes dislike of our own users ;)
                            > >
                            > > The real problem I want to solve is associating console output from a
                            > > running thread to a given test-case (for parallel runs). We have this
                            > > fairly well covered already and we manage to capture everything that
                            > > happens within a given test-case for that test-case.
                            > >
                            > > The real problem is that the RunListener API does not demarcate
                            > > test-classes properly, so I am unable to distinguish console output
                            > > happening in @AfterClass of one test with @BeforeClass on the test-class
                            > > on the same thread.
                            > >
                            > > This problem has been in the corner of my eye for quite some time and
                            > > I've been trying to find a decent angle to solve it from. I promise not
                            > > to do anything that is frowned upon, which is why I asked ;)
                            > >
                            > > Kristian
                            > >
                            >
                            > --
                            > Esko Luontola
                            > www.orfjackal.net
                            >
                            >
                          Your message has been successfully submitted and would be delivered to recipients shortly.