Loading ...
Sorry, an error occurred while loading the content.
 

Re: Order of execution for tests

Expand Messages
  • adrian.kuhn
    ... JExample is a test runner for JUnit that runs test in order of dependencies. You can specify that a test depends on the outcome of another test. What
    Message 1 of 38 , Jan 1, 2010
      --- In junit@yahoogroups.com, "yogesh.patil" <patilyogeshp@...> wrote:

      > I have a scenario to test where in I need to make sure that tests are
      > executed in particular order...
      > How can I achieve this?

      JExample is a test runner for JUnit that runs test in order of dependencies.
      You can specify that a test depends on the outcome of another test.

      What follows is considered heretic by some TDD proponents, but we believe that
      the burden of isolating tests should be shifted from developer to framework.

      We call this a producer - consumer relationship. The producer is always run
      before the consumer. When the producer fails, the consumer is skipped. We
      picked the names producer and consumer because the producer may actually
      *return* a value (ie produce a valid example of the class under test) and the
      consumer may take the value as a parameter (ie consumer the produced example).
      If two consumers depend on the same example, the JExample runner takes care to
      clone the example to avoid side-effects.

      The website is http://scg.unibe.ch/jexample

      Feedback and bug reports are welcome.

      PS, we also have a prototype that uses profiling to automatically insert the
      implicit dependencies of legacy tests. If you are interested, drop a mail.

      > Are the test cases added with annotation @Test gets executed in the order
      > they are written?

      The order is unspecified.

      cheers,
      AA
    • David Saff
      ... Rocco, To add a datapoint, working now in a rather large software engineering organization, I ve found that knowing which tests fail, even with
      Message 38 of 38 , Jan 9, 2010
        On Fri, Jan 8, 2010 at 1:41 PM, Pigneri, Rocco <rpigneri@...> wrote:
        > Dear Kent,
        >
        > I don’t think that I am actually the one to provide the answers.  I was simply providing a concrete example to James Lee’s regression situation (I was sorta drafted for it since I provided the 80-20 suggestion).  However, here’s a quick summary of where I see the real value-add for this feature: large enterprise customers who have large regression suites.  These are people who could use the dependencies to identify the cause of a regression failure and therefore pin-point the responsible party much more easily.  Also, since large companies usually have difficulty adopting new tools (i.e. politics, cost of training, etc.), they would benefit most from having this feature in JUnit itself rather than adopting a new tool.
        >
        > As a disclaimer, most of my corporate experience in is companies with fewer than 20 engineers: in these situations, you usually already know who to talk to ;-).  In fact, the example I gave isn’t even real; I just made it up!  However, I can speak from my limited experience to answer a few of these questions.  James, please provide your input as well since you have a more pressing “real-world” need.
        >
        > * What is the magnitude of the value? (That is, how long does it take to find the most-fundamental error when you have one test failing vs. when you have many?)
        > * How many people would use dependencies?
        >
        > I would also add “What kind of people would use dependencies?”
        >
        > I think our target audience here are regression suite maintenance engineers in large corporations.  With lots of moving parts, lots of engineers, and a large regression suite, the value of dependencies is incredible as the tests allow for automated identification of the group/person most responsible for breaking the build/test run.  You could easily imagine that for a project with 500 engineers letting the computer send an e-mail to the most responsible party based on the tests that fail could greatly reduce maintenance overhead for the regression engineers for transient errors.  This feature would then free the maintenance engineers to focus on code that seems constantly out of whack.  Add in the fact that a company of that size may have trouble adopting a new tool to automat this process, and you see that this provides a lot of value to large, corporate customers.

        Rocco,

        To add a datapoint, working now in a rather large software engineering
        organization, I've found that knowing which tests fail, even with
        sophisticated analysis, is rarely sufficient to find the people who
        broke the tests. Around here, team A doesn't often submit code that
        breaks team A's tests. Team A fixes all of team A's tests _before_
        submitting, and then the submitted code breaks team B's tests.

        David Saff

        >
        > * What is the cost of creating the dependencies?
        > * What is the cost of maintaining the dependencies?
        > * What is the cost of teaching about dependencies?
        >
        > I feel that these are rather minimal.  Discovering a missing dependency would simply involve gaining extraneous information about a failure rather than a succinct report, possibly leading to a few false leads for fixers.  The only time this would be expensive would be for classes shared over a large portion of a software project.  I’m not certain how often this situation would occur.
        >
        > * What is the cost to people who don't use dependencies?
        >
        > I would say none since they can just ignore this functionality, but I feel like there’s a nuance to this question that I am missing.
        >
        > * Do the dependencies ever provide misleading results? How frequently? At what cost?
        >
        > I have punted this idea around a lot as well.  Assuming that the dependencies are, in fact, correctly set, the only problem that I foresee is that a single failure in a common class could mask a lot of other, unrelated errors under the 80-20 solution because *any* dependent test fixture would be skipped for a single failure in a dependency test fixture.  However, this is one of the trade-offs of marking dependencies on the fixture level rather than the method level.
        >
        > * What is the cost and opportunity cost of both the initial implementation of dependencies and the maintenance of the code? (This is primarily a concern for David and I as our implementation budget for JUnit is so tight, but the opportunity cost is a factor for everyone.)
        >
        > Well, I can’t really speak to this since I haven’t developed for JUnit before and since I can’t see myself needing this feature in the near future, but I do have the following suggestion.  Since this feature really benefits our large enterprise customers and since this feature is being suggested in order to allow our corporate customers to avoid adding a new tool, I suggest the following approach: let’s hash out a spec—I think we are reasonably close to one already, but I will defer, again, to those with more experience in larger enterprises—and then state that we will support/maintain this feature if someone else is willing to write it, write it and maintain it for a year, or meet some other criteria whereby this feature is an even trade for our community.  Since we are considering this feature more out of concerns for corporate needs rather than out of a technical philosophy, I think that it is only fair for “corporate” to drive this feature’s implementation and help out a bit in its maintenance.
        >
        > Just my two cents,
        >
        > Rocco
        >
        > From: junit@yahoogroups.com [mailto:junit@yahoogroups.com] On Behalf Of Kent Beck
        > Sent: Wednesday, January 06, 2010 3:03 PM
        > To: junit@yahoogroups.com
        > Subject: Re: [junit] Order of execution for tests
        >
        >
        >
        > Rocco,
        >
        > Thank you for the complete and carefully considered example. I think from it I can understand the value you would like to gain from expressing dependencies. My questions are:
        > * How frequently does this value accrue?
        > * What is the magnitude of the value? (That is, how long does it take to find the most-fundamental error when you have one test failing vs. when you have many?)
        > * What is the cost of creating the dependencies?
        > * What is the cost of maintaining the dependencies?
        > * What is the cost of teaching about dependencies?
        > * What is the cost to people who don't use dependencies?
        > * How many people would use dependencies?
        > * Do the dependencies ever provide misleading results? How frequently? At what cost?
        > * What is the cost and opportunity cost of both the initial implementation of dependencies and the maintenance of the code? (This is primarily a concern for David and I as our implementation budget for JUnit is so tight, but the opportunity cost is a factor for everyone.)
        >
        > I don't necessarily expect "the answers" to these questions from you, but these are the things we're thinking about when we consider whether to implement any feature.
        >
        > Regards,
        >
        > Kent
        >
        > On Jan 6, 2010, at 9:47 AM, Pigneri, Rocco wrote:
        >
        >> Dear Kent,
        >>
        >> Let’s say that you are building a reporting tool for a financial firm and that you have three major indices to manage: consumer spending, average national income, and average consumer debt load. The calculation and presentation of each of these indices is rather involved, especially given the amount of background material required for regulatory compliance, so you have a separate three-tier class structure for each index. Each index has a few “helper classes” that are shared among the three indices and also between the logic and data layers: consumer, consumer region (a group of consumers within a particular region), and some analysis classes. On top of the presentation layer for each tier, there is a single “reporter” class that collates the three index reports, provides some summary data, and personalizes the package for the individual recipients.
        >>
        >> So, if you imagine that each of these parts is implemented in a single class—I’ve found this organization to be pretty reasonable in the past—you would have a few classes: Consumer, ConsumerRegion, ConsumerAnalysis at the bottom, then a single class for each tier of each index (SpendingDataLayer, SpendingLogicLayer, SpendingPresentationLayer, IncomeDataLayer, etc.), and then a final Reporter class at the top. The logic layers may also call upon the analysis classes for certain common statistical functions. The data layers would, in addition to poking directly into the database, utilize the “helper classes” for some basic DB access. You could also imagine having a single test class for each of the above classes (also reasonable in my experience).
        >>
        >> In JUnit, then, you would say that the ReporterTestFixture depends upon the SpendingPresentationLayerTestFixture, the IncomePresentationLayerTestFixture, and the DebtLoadPresentationLayerTestFixture, let’s say via an Annotation on ReporterTestFixture. Each presentation layer would be marked to depend upon its logic layer. The logics would depend upon the datas, any additional helper classes, and finally the Analysis class. Finally, the data layers would be marked to depend upon any “helper classes” that they use.
        >>
        >> Under this schema, if any of the DebtLoadLogicLayer’s tests were to fail, the DebtLoadPresentationLayerTestFixture and the ReporterTestFixture would be skipped, and the DebtLoadLogicLayerTestFixture’s tests would be highlighted as since they failed. The ConsumerSpendingLogic and PresentaionLayerTestFixtures would be run along with their analogues in Income as they do not depend upon the DebtLoadLogicLayer and report any errors found as described above. As another example, if any of Consumer’s tests fail, then nearly the entire tree would be skipped (all of the n-tier code plus the Reporter), thus showing only a failure in Consumer and maybe a failure in some other helper classes. In other words, you very quickly find the lowest level class where a failure occurred before it can percolate up through your test suite. This would greatly simplify life when running regression suites as the class that caused the error (along with its developer, hopefully) is easy identifiable.
        >>
        >> The major value-add of this approach is that you get most of the benefits of dependency trees without the hassle of maintaining thousands of dependency links. The class relationships above are rarely going to change—in other words, if the logic layer stops depending upon the data layer, you’ve probably rearchitected the system and are already being careful about what you are doing—and the annotations are very easy to write. Also, due to the hierarchical nature of the dependencies, you do have a bit of a safety net: if a Logic Layer forgets to claim a helper class dependency, but that Logic Layer’s Data Layer does claim that dependency, then the Logic Layer’s tests are still skipped when the helper class fails.
        >>
        >> The biggest shortcoming of this approach involves situations where you have lots of classes that all depend upon one helper class: for example, if we had thirty indices, we would have to maintain thirty annotations for each helper class on each data layer, thus requiring more maintenance to track the dependency. Another negative is that a single failing test in your test fixture could skip over 95% of your tests since this information is stored on a class rather than method level. However, the maintenance costs are much lower, and the method dependencies could be added later if we are smart about our framework.
        >>
        >> How does that sound?
        >>
        >> Thank you,
        >>
        >> Rocco
        >>
        >> From: junit@yahoogroups.com<mailto:junit%40yahoogroups.com> [mailto:junit@yahoogroups.com<mailto:junit%40yahoogroups.com>] On Behalf Of Kent Beck
        >> Sent: Tuesday, January 05, 2010 1:07 PM
        >> To: junit@yahoogroups.com<mailto:junit%40yahoogroups.com>
        >> Subject: Re: [junit] Order of execution for tests
        >>
        >>
        >>
        >> Could you provide a concrete example so I'm sure I understand what you're saying?
        >>
        >> Thanks,
        >>
        >> Kent
        >>
        >> On Jan 5, 2010, at 8:21 AM, Pigneri, Rocco wrote:
        >>
        >> > Dear all,
        >> >
        >> > An interesting 80-20 solution for this scenario could be something as simple as an attribute indicating that a test fixture depends upon the passing of all tests in another fixture. This would allow for failure scenarios to be on the scale of Cédric’s “10 tests failed” for James’s large regression suite without requiring a lot of effort from the user-developers. What makes this particularly nice in my mind is that you only have to change these dependencies when you rearchitect the system, thus making them rather light on the maintenance side.
        >> >
        >> > What do you all think?
        >> >
        >> > Rocco
        >> >
        >> > From: junit@yahoogroups.com<mailto:junit%40yahoogroups.com><mailto:junit%40yahoogroups.com> [mailto:junit@yahoogroups.com<mailto:junit%40yahoogroups.com><mailto:junit%40yahoogroups.com>] On Behalf Of James Lee
        >> > Sent: Monday, January 04, 2010 3:43 PM
        >> > To: junit@yahoogroups.com<mailto:junit%40yahoogroups.com><mailto:junit%40yahoogroups.com>
        >> > Subject: Re: [junit] Order of execution for tests
        >> >
        >> >
        >> >
        >> > Or in my case, something like this:
        >> >
        >> > 1 test failed, 585 tests skipped (how it would have looked in a recent test
        >> > run).
        >> >
        >> > Our suite has well over 5000 tests and couple dozen developers all
        >> > adding/changing in parallel.
        >> >
        >> > Luckily we use a continuous integration environment, so get a small set of
        >> > changes for every run (run the tests every 2hrs). So it can usually be
        >> > tracked down in 20-30m of work. But it would be almost instant if there was
        >> > a "core" test that failed first. Or even if it wasn't perfect, like: 25
        >> > tests failed, 585 tests skipped.
        >> >
        >> > Gets much worse when the folks watching the builds are busy, as it can fail
        >> > for long enough to cloud the build history, as failure states start to
        >> > overlap.
        >> >
        >> > Just throwing in the, "Crazy large test suite" perspective.
        >> >
        >> > Regards,
        >> > James Lee
        >> >
        >> > 2010/1/4 Cédric Beust ♔ <cbeust@...<mailto:cbeust%40google.com><mailto:cbeust%40google.com><mailto:cbeust%40google.com>>
        >> >
        >> > >
        >> > >
        >> > > On Mon, Jan 4, 2010 at 11:25 AM, Charlie Poole
        >> > > <charlie@...<mailto:charlie%40pooleconsulting.com><mailto:charlie%40pooleconsulting.com><mailto:charlie%40pooleconsulting.com> <charlie%40pooleconsulting.com>>wrote:
        >> > >
        >> > > > And yet, I get reports of other folks having this interpretation
        >> > > > problem all the time. It may be that the problem lies in how they
        >> > > > are doing their development.
        >> > > >
        >> > >
        >> > > I don't think so, it's just that
        >> > >
        >> > > "1 test failed, 9 tests skipped"
        >> > >
        >> > > is much easier to read and diagnose than
        >> > >
        >> > > "10 tests failed".
        >> > >
        >> > > --
        >> > > ***Cédric
        >> > > *
        >> > >
        >> > > [Non-text portions of this message have been removed]
        >> > >
        >> > >
        >> > >
        >> >
        >> > [Non-text portions of this message have been removed]
        >> >
        >> >
        >> > [Non-text portions of this message have been removed]
        >> >
        >> >
        >>
        >> [Non-text portions of this message have been removed]
        >>
        >>
        >> [Non-text portions of this message have been removed]
        >>
        >>
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.