Loading ...
Sorry, an error occurred while loading the content.

[XP] Testing Multi threaded programs

Expand Messages
  • Keith Nicholas
    Hi all, what do people do to unit test multithreaded software in C++? How do you test critical sections etc? how would I prove that I got my critical section
    Message 1 of 6 , Apr 2, 2000
    • 0 Attachment
      Hi all,

      what do people do to unit test multithreaded software in C++? How do you
      test critical sections etc? how would I prove that I got my critical
      section right? etc etc...

      Regards,

      Keith
    • c@nautronix.com.au
      =From: Keith Nicholas = =Hi all, = =what do people do to unit test multithreaded software in C++? How do you well i dunno what The
      Message 2 of 6 , Apr 3, 2000
      • 0 Attachment
        =From: "Keith Nicholas" <keithn@...>
        =
        =Hi all,
        =
        =what do people do to unit test multithreaded software in C++? How do you

        well i dunno what The Word(tm) is, but i do it like this ...

        make each class you write as thread safe as you know how.
        each class has a test harness which tests that class (btw this writing
        the test harness first is a great idea, feels weird to start with but
        it does make things easier, tho i'm not quite sure how/why yet).
        refactor till the test harness works.
        repeat.

        this way you dont get all the multithreading bugs immediately but you
        do get them.

        when i get to a high level test harness which is testing most of the system
        i'll put it in a loop and leave it running all night. its also usefull
        to arbitrarily change the load on the machine (both up and down) to
        shake out any weirdness.

        however i have been wondering if XP would advocate trying to coming
        up with more strenuous testing of multithreadedness at the lower levels?
        is it worth the effort? my gut reaction is that it isnt as you will exercise
        the system more and more as you get to the higher level test harnesses.
        comments?
        c


        =test critical sections etc? how would I prove that I got my critical
        =section right? etc etc...
        =
        =Regards,
        =
        =Keith

        ps for those doing multithreaded programming the newsgroup
        comp.programming.multithreaded is well worth reading especially the faqs
        and the ideas on using thread pools.
      • Chris Bitmead
        ... This is another argument for automated tests. If you have automated tests you can just run 10 threads doing the same thing, which is often a good general
        Message 3 of 6 , Apr 3, 2000
        • 0 Attachment
          c@... wrote:

          > however i have been wondering if XP would advocate trying to coming
          > up with more strenuous testing of multithreadedness at the lower levels?
          > is it worth the effort? my gut reaction is that it isnt as you will exercise
          > the system more and more as you get to the higher level test harnesses.
          > comments?

          This is another argument for automated tests. If you have automated
          tests you can
          just run 10 threads doing the same thing, which is often a good general
          test
          of threads.
        • James W Grenning
          ... The team I am coaching is developing a multi-threaded c++. This application will run in NT but we are designing it to be portable to other platforms by
          Message 4 of 6 , Apr 4, 2000
          • 0 Attachment
            > Keith wrote:
            > what do people do to unit test multithreaded software in C++? How do you
            > test critical sections etc? how would I prove that I got my critical
            > section right? etc etc...
            >
            > c@... wrote:
            >
            > > however i have been wondering if XP would advocate trying to coming
            > > up with more strenuous testing of multithreadedness at the lower levels?
            > > is it worth the effort? my gut reaction is that it isnt as you
            > will exercise
            > > the system more and more as you get to the higher level test harnesses.
            > > comments?
            >
            > This is another argument for automated tests. If you have automated
            > tests you can
            > just run 10 threads doing the same thing, which is often a good general
            > test
            > of threads.
            >

            The team I am coaching is developing a multi-threaded c++. This application
            will run in NT but we are designing it to be portable to other platforms by
            using ACE (Doug Schmidt's asynchronous toolkit). The end application will
            hold about 50 copies of a server, each server will need about 4 threads to
            do its job. This application will probable have a couple hundred threads.

            Our testing approach has been to start out treating the system as if it has
            only one thread and getting the simplified single threaded application to
            perform some core functions. We have automated one set of tests for the
            application as if it were single threaded. We use CppUnit. These tests
            check the system against its basic requirements. We make use of abstract
            interfaces to give us test and simulation points. Imagine a layered part of
            the system. To test a middle layer, a simulated bottom layer is provided.
            The test case drives the middle layer and get feedback from the simulated
            bottom layer. The middle layer is happily tested as if were running in the
            real system.

            After having basic functionality going, our next release added the
            complicating issues dealing with the first level of threading needed.
            Specifically we are still providing a single server, but it includes
            multiple threads for concurrent operations. The asynchronous operations
            have been implementing using a variation of the Active Object pattern. With
            this pattern, the tasking issues can be separated (somewhat) from the
            application logic. The Active Object serializes access to some critical
            resources, thereby lessening the need for specific critical sections. This
            lightly multi-threaded application is tested again with CppUnit.

            Because the threading issues are separated from the application logic, the
            application logic can be tested in isolation from the threads. We have
            found this to be very valuable.

            The next level of design and test is to replicate the key objects 50 times,
            and feed transactions into the system, load testing the entire application.
            Transaction scenarios are designed and written to stress the system. When
            bugs occur, the repeatable test sequence is very valuable.

            Regarding critical sections, critical section constructs can be unit tested,
            but the placement of critical section operating in your code is probably
            best verified through inspection and pair programming. An object that can
            be accessed by multiple threads has to have shared member variables
            protected. This can be inspected. A design where critical resources are
            only accessed serially through an Active Object can cut down on the
            opportunities for concurrency problems. A load test will help find the ones
            that sneak through.

            James

            -------------------------------------------------------------
            James Grenning High Point Software Technology Company
            Design Consulting 10 High Point Drive (847) 438-9942
            Project Management Hawthorn Woods, IL
            OO Training 60047
            jgrenning@... www.HighPointSoftware.com
            --------------------------------------------------------------
          • Baker, Bram
            My 2 cents, If you can, try running some of your load tests on a multi-processor machine. This can drive out some of the really short race conditions you might
            Message 5 of 6 , Apr 5, 2000
            • 0 Attachment
              RE: [XP] Testing Multi threaded programs

              My 2 cents,

              If you can, try running some of your load tests on a multi-processor machine. This can drive out some of the really short race conditions you might have in your code. May also reveal some deadlock scenarios, but usually not ... these can be found easily enough through load testing on a single processor machine. I have found that testing on a multi-processor is beneficial. I have also been in situations were some code I was involved with worked fine on an installed base of 50,000 uni-processor machines, but died horribly on the only multi-processor machine our code was installed on (it turned out to be a bug of sorts in the OS).

              Bram

              PS. if you're using DCOM on NT, make sure you've applied NTsp5 or greater. There's some thread safety issues in the DCOM marshalling code (really in the RPC layer) in NT sp4 and lower. No clue about Win9x.

              -----Original Message-----
              From: James W Grenning [mailto:jgrenning@...]
              Sent: 2000, April, 04 11:39 PM
              To: extremeprogramming@egroups.com
              Subject: RE: [XP] Testing Multi threaded programs


              > Keith wrote:
              > what do people do to unit test multithreaded software in C++?  How do you
              > test critical sections etc?  how would I prove that I got my critical
              > section right? etc etc...
              >
              > c@... wrote:
              >
              > > however i have been wondering if XP would advocate trying to coming
              > > up with more strenuous testing of multithreadedness at the lower levels?
              > > is it worth the effort?  my gut reaction is that it isnt as you
              > will exercise
              > > the system more and more as you get to the higher level test harnesses.
              > > comments?
              >
              > This is another argument for automated tests. If you have automated
              > tests you can
              > just run 10 threads doing the same thing, which is often a good general
              > test
              > of threads.
              >

              The team I am coaching is developing a multi-threaded c++.  This application
              will run in NT but we are designing it to be portable to other platforms by
              using ACE (Doug Schmidt's asynchronous toolkit).  The end application will
              hold about 50 copies of a server, each server will need about 4 threads to
              do its job.  This application will probable have a couple hundred threads.

              Our testing approach has been to start out treating the system as if it has
              only one thread and getting the simplified single threaded application to
              perform some core functions.  We have automated one set of tests for the
              application as if it were single threaded.  We use CppUnit.  These tests
              check the system against its basic requirements.  We make use of abstract
              interfaces to give us test and simulation points.  Imagine a layered part of
              the system.  To test a middle layer, a simulated bottom layer is provided.
              The test case drives the middle layer and get feedback from the simulated
              bottom layer. The middle layer is happily tested as if were running in the
              real system.

              After having basic functionality going, our next release added the
              complicating issues dealing with the first level of threading needed.
              Specifically we are still providing a single server, but it includes
              multiple threads for concurrent operations.  The asynchronous operations
              have been implementing using a variation of the Active Object pattern.  With
              this pattern, the tasking issues can be separated (somewhat) from the
              application logic.  The Active Object serializes access to some critical
              resources, thereby lessening the need for specific critical sections.  This
              lightly multi-threaded application is tested again with CppUnit.

              Because the threading issues are separated from the application logic, the
              application logic can be tested in isolation from the threads.  We have
              found this to be very valuable.

              The next level of design and test is to replicate the key objects 50 times,
              and feed transactions into the system, load testing the entire application.
              Transaction scenarios are designed and written to stress the system.  When
              bugs occur, the repeatable test sequence is very valuable.

              Regarding critical sections, critical section constructs can be unit tested,
              but the placement of critical section operating in your code is probably
              best verified through inspection and pair programming.  An object that can
              be accessed by multiple threads has to have shared member variables
              protected.  This can be inspected.  A design where critical resources are
              only accessed serially through an Active Object can cut down on the
              opportunities for concurrency problems.  A load test will help find the ones
              that sneak through.

              James

              -------------------------------------------------------------
              James Grenning      High Point Software Technology Company
              Design Consulting   10 High Point Drive  (847) 438-9942
              Project Management  Hawthorn Woods, IL
              OO Training         60047
              jgrenning@...     www.HighPointSoftware.com
              --------------------------------------------------------------



              To Post a message, send it to:   extremeprogramming@...

              To Unsubscribe, send a blank message to: extremeprogramming-unsubscribe@...

              Ad-free courtesy of objectmentor.com

            • William Rutiser
              Test with more than two concurrent things( threads or processes ). Twenty years ago, I learned, the hard way, that carefully designed and formally proven
              Message 6 of 6 , Apr 5, 2000
              • 0 Attachment
                Test with more than two concurrent things( threads or processes ).

                Twenty years ago, I learned, the hard way, that carefully designed
                and formally proven interlock schemes can be correct for two
                processes and still fail for three.

                Bill Rutiser
              Your message has been successfully submitted and would be delivered to recipients shortly.