Loading ...
Sorry, an error occurred while loading the content.

Testing Concurrency, Was: Re: [XP] Kent Beck says write tests when they make sense (was Kent Beck says TDD is optional for short-haul work)

Expand Messages
  • Manuel Klimek
    To jump in on the testing concurrency topic, since I m currently writing a distributed and concurrent system in a what I d call test driven way, here s some
    Message 1 of 1 , May 19, 2009
      To jump in on the "testing concurrency" topic, since I'm currently writing a
      distributed and concurrent system in a what I'd call test driven way, here's
      some of what I learned:

      I have tests that are called like *ShouldBeThreadSafe, which are just
      calling different methods from lots of threads in parallel. I'd not call
      them unit tests, but they're testing what I really want - I want the class
      to be thread safe even in situations that I don't think about. The cost is
      usually about 2-10 seconds of runtime, and errors may not be detected "at
      once". As such they're not perfect tests, but they're very easy to write and
      for me they usually provide a high value. They're automated and I can write
      them to "drive" my concurrency statements because they usually give me the
      error I want with a high probability. They're not flaky because they cannot
      give an error when there is none, but they're probabilistic in that they
      only show me the error with a certain probability - it could be I miss an

      If I have a multithreaded problem, I write a unit tests that exactly
      reproduces the problem by somehow injecting notify- and wait statements. The
      cost for writing those tests is exceptionally high, because it's very hard
      to get the test correct in the first place - using them to test drive the
      concurrency statements in the code is practically impossible (for me ;) If I
      write a component together with those tests, I usually write the test, it
      fails. I write the code, the test still fails. It takes some time until I
      understand why the test doesn't work, and I fix the test. I uncomment the
      production code again, to see that the test fails. I reapply the code to
      make the test pass again. Most of the time writing the code is an order of
      magnitude simpler than writing the test.

      For some locking I rely on my system level tests to catch errors. I spent a
      lot of time setting up a test system that actually represents a "real"
      system as closely as possible, that runs automatically and takes about 8-10
      minutes to run, while it really does ugly stuff with the system. Those are
      the most valuable tests for detecting errors for me, because many of the
      components are very simple on a unit level, but the interaction gets really
      hard to wrap your head around. In this case I sometimes don't write tests
      for the basic locking, because the value I get from it (I fricking know this
      does not break) does not offset the cost (a either long running or hard to
      write test).

      Some more stuff on this topic that I found helpful:

      Who else has experience with testing concurrent or distributed systems?
      Ideas? Comments?


      On Tue, May 19, 2009 at 5:15 PM, kentb <kentb@...> wrote:

      > Phil,
      > I didn't intend to say that startup projects don't need tests. I have quite
      > a few tests and I absolutely rely on them. If tests are easy to write I
      > always write them. Generally, if I know how to write a test they are easy
      > to
      > write. However, because the budget is so tight while Max is still on the
      > runway, sometimes I have to choose between learning something about testing
      > and learning something about my market. In such a case it's no contest:
      > learn about the market. If I don't learn about my market, the product dies.
      > This came up today again. I was working on handling large workspaces
      > efficiently and I ran into concurrency problems. I'm sure there are some
      > really good ways to test concurrency out there, but I don't know any of
      > them. I tested manually as well as I could and released. I want to find out
      > if people with larger projects are more likely to buy and recommend Max if
      > it works better for them. I wish I had a button to push that would say,
      > "Nope, no concurrency problems here," but in the absence of such a button I
      > still have a product to get off the ground.
      > Regarding using JUnit to market Max, I don't have any good ideas of how to
      > do that. HotMail had this lovely feature where every message carried a
      > footer that invited people to sign up for HotMail. Since people interact
      > with JUnit mostly through IDEs or automated tasks, I can't think of how to
      > insert such a message at all, much less in a non-annoying way.
      > Thanks for the description of your buying process. It's hard to get
      > feedback, especially from the 98% of people who don't buy Max after looking
      > at the subscription page.
      > Your bond idea sounds intriguing, but I'm not sure how to pull it off. If
      > you could get people to buy the bonds it would give you a large community
      > of
      > people who were aligned with your success. Indulging in a flight of fancy
      > for a moment it seems like you could have voting and non-voting bonds,
      > where
      > the voting bonds had a say in weekly planning. Securitization is out of
      > vogue at the moment, but it might provide a good model for financing
      > software in a post-license-revenue world.
      > Regards,
      > Kent Beck
      > Three Rivers Institute
      > _____
      > From: extremeprogramming@yahoogroups.com<extremeprogramming%40yahoogroups.com>
      > [mailto:extremeprogramming@yahoogroups.com<extremeprogramming%40yahoogroups.com>]
      > On Behalf Of Phil Goodwin
      > Sent: Monday, May 18, 2009 11:13 PM
      > To: extremeprogramming@yahoogroups.com<extremeprogramming%40yahoogroups.com>
      > Subject: Re: [XP] Kent Beck says write tests when they make sense (was Kent
      > Beck says TDD is optional for short-haul work)
      > > Regarding the comments in my blog post: I'm in no rush. If you'd like to
      > > read it before discussing it I'll be glad to wait.
      > Sounds like I'm off base here. I did read it -- the post I mean, not
      > the reader comments -- but maybe I read some things into it that
      > weren't there. I hope I haven't made too much of an ass of myself...
      > Ok, I see what happened here. I took what I remembered of your post
      > and extrapolated based on earlier emails in this thread. The take-away
      > I'm getting is that you aren't keeping very much technical debt around
      > at all, and what you are allowing in is scattered more or less evenly
      > and so won't turn into big clumps of impenetrable code. I had
      > envisioned larger pieces resisting automated unit tests so that you
      > were doing something else that wasn't leading to a regression suite. I
      > also became overly fond of the bit about untested legacy code and the
      > grad school dropout (it originally appeared as an obvious
      > exaggeration), I hope it wasn't offensive.
      > One interesting thing is that you spent a whole week just on your
      > first test. So it's not a case of being unwilling to spend extra time
      > on testing at all, but one where it occasionally seems too costly to
      > create a single one-off test of something relatively obscure, and for
      > which you have alternate sources of error reporting anyway.
      > > Regarding getting paid to write tools: JUnit Max *is* my upsell from
      > JUnit.
      > > If you like the feedback you get from JUnit, Max gives you more feedback,
      > > quicker, easier, and with less distraction.
      > But JUnit isn't acting as a marketing tool for JUnit Max. If by using
      > JUnit I was exposed to information about the availability and feature
      > set of JUnit Max, I would have been much more likely to have paid for
      > it before now. Acrobat and Quicktime both use this technique -- to
      > excess, actually -- but I think that it's something that could be done
      > in a tasteful, unobtrusive way, so that users would always know that
      > there is an upgrade option on how to access it, without it being
      > shoved in their faces all the time.
      > > Regarding the $2/month price: the first question I needed to answer was,
      > > "Will anyone pay anything for a tool like this?" $2 was a symbolic amount
      > to
      > > validate the feedback. The answer, BTW, is yes, some people will pay
      > > something for a tool like Max. Now the question is, "Will enough people
      > pay
      > > enough to create a viable business?" For that I will have to slowly raise
      > > the price to determine the commercial price the market is willing to pay.
      > I wonder how the psychology of that will play out. For some things you
      > can start the price out high and then drop it gradually to find what
      > the market will bear -- and end up with a relatively high price
      > because of the initial perception of high value. I don't have a sense
      > of whether such a thing would work for software though.
      > > There was also a naive, self-effacing aspect to the $2/month price. I
      > wanted
      > > to set the price so low that no one would have to be without Max if they
      > > wanted it. I quickly found out there is no price > 0 that meets this
      > > criterion.
      > Yes, there is a hurdle to getting people to spend any money at all.
      > For me this has been raised much higher by all the hidden fees and
      > opt-out gimmicks I've run into over the years. I just paid for Max
      > this evening and my perception, for what it's worth, is that I spent
      > about $25 for this year's version of it. Which is to say, I spent $24
      > in a lump and that gave me the feeling that I would be entitled to use
      > the most recent version for as long as I like and that I would receive
      > access to all upgrades for the next 12 months. When I contemplated
      > paying $2 for a single month I assumed that I would lose my right to
      > use it at all after that month. I put virtually no conscious thought
      > into any of this and freely admit that it is completely irrational --
      > it's just the impression I gleaned as I was completing the payment
      > process for the impulse purchase I made on my way to your blog.
      > I wonder if people would behave differently if you sold JUnit Max,
      > along with a year's worth of free upgrades for the low price of just
      > $24 and then offered a subscription to subsequent upgrades for only
      > $2/mo. Maybe computer nerds would just filter out the marketing-speak
      > and treat it exactly the same.
      > > Your idea about funding free software development is interesting. I
      > wouldn't
      > > mind making a commitment to open sourcing Max after $X dollars of
      > revenue.
      > > Is that the sort of thing you're thinking of?
      > That would be part of it. The idea is to finance initial development
      > with a loan and to use the IP to be developed as collateral for that
      > loan. I was thinking that it would be easier and safer for the
      > developer to get lots of little loans instead of one big one, hence
      > the idea of selling bonds. So maybe you sell 20,000 bonds for $50 each
      > in order to raise $100,000 for initial development. Say that you want
      > to offer 10% interest so you make each one pay $0.00275 each for the
      > first 20,000 units shipped ($0.00275 * 20,000 = $55), then you sell
      > each unit for $60. The commitment to open source increases the utility
      > value of the end product. Presumably the bond holders had an interest
      > in bringing the finished product into the world. Open sourcing it
      > gives it the best chance of remaining available and viable after the
      > initial development and quells a potential source of squabbling among
      > the bond holders. I don't see it as necessary, but it seems like a
      > good idea to me.
      > Phil
      > [Non-text portions of this message have been removed]


      [Non-text portions of this message have been removed]
    Your message has been successfully submitted and would be delivered to recipients shortly.