Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Real or Unreal Test. Help me argue with my colleagues

Expand Messages
  • Curtis Cooley
    ... In my opinion, acceptance testing should include as much of the real world as practical. These are tests that don t necessarily have to run fast. They
    Message 1 of 7 , Apr 26, 2011
    • 0 Attachment
      On Apr 26, 2011 7:00 AM, "Steven Woody" <narkewoody@...> wrote:
      >
      >
      >
      > Hi,
      >
      > Our product is an embedded device. Looking externally, it is a
      > powered box, say A. On one side of the box, their is a USB interface
      > that need to be connected with another device B. Inside the box, there
      > exists a wireless modem chip C. Our system, basically, is to read
      > values from A, and make it readable via the wireless modem from a
      > client program running on PC.
      >
      > User cannot directly play with the box A, he has to operate with
      > device B and play with a PC read program on a PC.
      >
      > Now, we concern how the acceptance test for our box A should be done.
      > My options is to mock device A and modem C, since I think the system
      > under test is just A. But many colleagues think this kind of test is
      > unreal and want to put the real device B and real modem C together in
      > a test setup.
      >
      > My reasoning has not yet satisfying them so much. Would you please
      > tell me your opinion and help me in argue with them?
      >

      In my opinion, acceptance testing should include as much of the 'real world'
      as practical. These are tests that don't necessarily have to run fast. They
      are your regression suite that tells you when something is broken.

      A test/qa lab with real world equipment will provide the highest level of
      confidence in the system.


      [Non-text portions of this message have been removed]
    • nader1talai@gmail.com
      Here is my understanding of the problem and suggestion Input is entered via B, A reads values and makes the values available to C C transmits values over the
      Message 2 of 7 , Apr 26, 2011
      • 0 Attachment
        Here is my understanding of the problem and suggestion

        Input is entered via B,
        A reads values and makes the values available to C
        C transmits values over the air to PC.

        I suggest that you need tests at system the whole end to end, module A, B, C and unit test level.

        You may build up the end to end tests through mocks and you will also need end to end test with the real system.

        Nader
        Sent from my BlackBerry� wireless device

        -----Original Message-----
        From: Steven Woody <narkewoody@...>
        Sender: extremeprogramming@yahoogroups.com
        Date: Tue, 26 Apr 2011 22:00:11
        To: <extremeprogramming@yahoogroups.com>
        Reply-To: extremeprogramming@yahoogroups.com
        Subject: [XP] Real or Unreal Test. Help me argue with my colleagues

        Hi,

        Our product is an embedded device. Looking externally, it is a
        powered box, say A. On one side of the box, their is a USB interface
        that need to be connected with another device B. Inside the box, there
        exists a wireless modem chip C. Our system, basically, is to read
        values from A, and make it readable via the wireless modem from a
        client program running on PC.

        User cannot directly play with the box A, he has to operate with
        device B and play with a PC read program on a PC.

        Now, we concern how the acceptance test for our box A should be done.
        My options is to mock device A and modem C, since I think the system
        under test is just A. But many colleagues think this kind of test is
        unreal and want to put the real device B and real modem C together in
        a test setup.

        My reasoning has not yet satisfying them so much. Would you please
        tell me your opinion and help me in argue with them?

        Thanks in advance.


        --
        Life is the only flaw in an otherwise perfect nonexistence
        � � -- Schopenhauer

        narke
        public key at http://subkeys.pgp.net:11371 (narkewoody@...)



        [Non-text portions of this message have been removed]
      • Steven Woody
        ... Thanks for the opinion. Maybe not to call it acceptance test, but system test? I am not so familiar to these definitions. The problem here is How to
        Message 3 of 7 , Apr 26, 2011
        • 0 Attachment
          On 26 April 2011 22:09, Curtis Cooley <curtis.cooley@...> wrote:

          >
          >
          > On Apr 26, 2011 7:00 AM, "Steven Woody" <narkewoody@...> wrote:
          > >
          > >
          > >
          > > Hi,
          > >
          > > Our product is an embedded device. Looking externally, it is a
          > > powered box, say A. On one side of the box, their is a USB interface
          > > that need to be connected with another device B. Inside the box, there
          > > exists a wireless modem chip C. Our system, basically, is to read
          > > values from A, and make it readable via the wireless modem from a
          > > client program running on PC.
          > >
          > > User cannot directly play with the box A, he has to operate with
          > > device B and play with a PC read program on a PC.
          > >
          > > Now, we concern how the acceptance test for our box A should be done.
          > > My options is to mock device A and modem C, since I think the system
          > > under test is just A. But many colleagues think this kind of test is
          > > unreal and want to put the real device B and real modem C together in
          > > a test setup.
          > >
          > > My reasoning has not yet satisfying them so much. Would you please
          > > tell me your opinion and help me in argue with them?
          > >
          >
          > In my opinion, acceptance testing should include as much of the 'real
          > world'
          > as practical. These are tests that don't necessarily have to run fast. They
          > are your regression suite that tells you when something is broken.
          >
          > A test/qa lab with real world equipment will provide the highest level of
          > confidence in the system.
          >
          >
          Thanks for the opinion. Maybe not to call it acceptance test, but system
          test? I am not so familiar to these definitions.

          The problem here is "How to control the real world in the terms of
          testing?". Making a value change to the device B can sometimes be a very
          hard job in our application, but to device A, we only need to ensure it can
          read a value back from B. If extern to the wireless mode, things can be
          even impossible, since there is no idea to control the GSM/GPRS network.



          > [Non-text portions of this message have been removed]
          >
          >
          >



          --
          Life is the only flaw in an otherwise perfect nonexistence
          -- Schopenhauer

          narke
          public key at http://subkeys.pgp.net:11371 (narkewoody@...)


          [Non-text portions of this message have been removed]
        • Steven Smith
          I think you re both right. In any system, you want to have a mixture of unit, integration, and acceptance tests. The exact mix of how many of these you d
          Message 4 of 7 , Apr 26, 2011
          • 0 Attachment
            I think you're both right. In any system, you want to have a mixture of
            unit, integration, and acceptance tests. The exact mix of how many of these
            you'd like to have (and how many you actually end up writing) will vary
            based on many factors, such as your team's experience with testing (and each
            kind of test) and the details of the application itself. Generally, you
            want to write as many tests as possible at the unit and integration end of
            the scale, as these tend to run faster and break less frequently than
            acceptance tests, which in many cases must be done manually in addition to
            having many more dependencies. I think the best way to approach the
            argument with your peers is to accept their side of the argument, but make
            the point that this kind of testing is a "defense in depth" approach, and
            there are things you could be catching with your unit and integration tests
            more easily and cheaply than with the acceptance tests on real hardware.

            Steve


            On Tue, Apr 26, 2011 at 10:00 AM, Steven Woody <narkewoody@...> wrote:

            >
            >
            > Hi,
            >
            > Our product is an embedded device. Looking externally, it is a
            > powered box, say A. On one side of the box, their is a USB interface
            > that need to be connected with another device B. Inside the box, there
            > exists a wireless modem chip C. Our system, basically, is to read
            > values from A, and make it readable via the wireless modem from a
            > client program running on PC.
            >
            > User cannot directly play with the box A, he has to operate with
            > device B and play with a PC read program on a PC.
            >
            > Now, we concern how the acceptance test for our box A should be done.
            > My options is to mock device A and modem C, since I think the system
            > under test is just A. But many colleagues think this kind of test is
            > unreal and want to put the real device B and real modem C together in
            > a test setup.
            >
            > My reasoning has not yet satisfying them so much. Would you please
            > tell me your opinion and help me in argue with them?
            >
            > Thanks in advance.
            >
            > --
            > Life is the only flaw in an otherwise perfect nonexistence
            > -- Schopenhauer
            >
            > narke
            > public key at http://subkeys.pgp.net:11371 (narkewoody@...)
            >
            >



            --
            Steve Smith
            http://SteveSmithBlog.com/
            http://twitter.com/ardalis


            [Non-text portions of this message have been removed]
          • Curtis Cooley
            ... This characterizes what I meant, but much more clearly. Thanks Steven. -- Curtis Cooley curtis.cooley@gmail.com home:http://curtiscooley.com
            Message 5 of 7 , Apr 26, 2011
            • 0 Attachment
              On Tue, Apr 26, 2011 at 7:38 AM, Steven Smith <ssmith.lists@...> wrote:
              > I think you're both right.  In any system, you want to have a mixture of
              > unit, integration, and acceptance tests.  The exact mix of how many of these
              > you'd like to have (and how many you actually end up writing) will vary
              > based on many factors, such as your team's experience with testing (and each
              > kind of test) and the details of the application itself.  Generally, you
              > want to write as many tests as possible at the unit and integration end of
              > the scale, as these tend to run faster and break less frequently than
              > acceptance tests, which in many cases must be done manually in addition to
              > having many more dependencies.  I think the best way to approach the
              > argument with your peers is to accept their side of the argument, but make
              > the point that this kind of testing is a "defense in depth" approach, and
              > there are things you could be catching with your unit and integration tests
              > more easily and cheaply than with the acceptance tests on real hardware.
              >

              This characterizes what I meant, but much more clearly. Thanks Steven.
              --
              Curtis Cooley
              curtis.cooley@...
              home:http://curtiscooley.com
              blog:http://ponderingobjectorienteddesign.blogspot.com
              ===============
              Leadership is a potent combination of strategy and character. But if
              you must be without one, be without the strategy.
              -- H. Norman Schwarzkopf
            • Steven Woody
              Thanks a lot!! I basically think same with you, but your language is so much better! On Apr 26, 2011 10:39 PM, Steven Smith wrote:
              Message 6 of 7 , Apr 26, 2011
              • 0 Attachment
                Thanks a lot!! I basically think same with you, but your language is so much
                better!
                On Apr 26, 2011 10:39 PM, "Steven Smith" <ssmith.lists@...> wrote:


                [Non-text portions of this message have been removed]
              Your message has been successfully submitted and would be delivered to recipients shortly.