Loading ...
Sorry, an error occurred while loading the content.

Real or Unreal Test. Help me argue with my colleagues

Expand Messages
  • Steven Woody
    Hi, Our product is an embedded device. Looking externally, it is a powered box, say A. On one side of the box, their is a USB interface that need to be
    Message 1 of 7 , Apr 26, 2011
    • 0 Attachment
      Hi,

      Our product is an embedded device. Looking externally, it is a
      powered box, say A. On one side of the box, their is a USB interface
      that need to be connected with another device B. Inside the box, there
      exists a wireless modem chip C. Our system, basically, is to read
      values from A, and make it readable via the wireless modem from a
      client program running on PC.

      User cannot directly play with the box A, he has to operate with
      device B and play with a PC read program on a PC.

      Now, we concern how the acceptance test for our box A should be done.
      My options is to mock device A and modem C, since I think the system
      under test is just A. But many colleagues think this kind of test is
      unreal and want to put the real device B and real modem C together in
      a test setup.

      My reasoning has not yet satisfying them so much. Would you please
      tell me your opinion and help me in argue with them?

      Thanks in advance.


      --
      Life is the only flaw in an otherwise perfect nonexistence
          -- Schopenhauer

      narke
      public key at http://subkeys.pgp.net:11371 (narkewoody@...)
    • Curtis Cooley
      ... In my opinion, acceptance testing should include as much of the real world as practical. These are tests that don t necessarily have to run fast. They
      Message 2 of 7 , Apr 26, 2011
      • 0 Attachment
        On Apr 26, 2011 7:00 AM, "Steven Woody" <narkewoody@...> wrote:
        >
        >
        >
        > Hi,
        >
        > Our product is an embedded device. Looking externally, it is a
        > powered box, say A. On one side of the box, their is a USB interface
        > that need to be connected with another device B. Inside the box, there
        > exists a wireless modem chip C. Our system, basically, is to read
        > values from A, and make it readable via the wireless modem from a
        > client program running on PC.
        >
        > User cannot directly play with the box A, he has to operate with
        > device B and play with a PC read program on a PC.
        >
        > Now, we concern how the acceptance test for our box A should be done.
        > My options is to mock device A and modem C, since I think the system
        > under test is just A. But many colleagues think this kind of test is
        > unreal and want to put the real device B and real modem C together in
        > a test setup.
        >
        > My reasoning has not yet satisfying them so much. Would you please
        > tell me your opinion and help me in argue with them?
        >

        In my opinion, acceptance testing should include as much of the 'real world'
        as practical. These are tests that don't necessarily have to run fast. They
        are your regression suite that tells you when something is broken.

        A test/qa lab with real world equipment will provide the highest level of
        confidence in the system.


        [Non-text portions of this message have been removed]
      • nader1talai@gmail.com
        Here is my understanding of the problem and suggestion Input is entered via B, A reads values and makes the values available to C C transmits values over the
        Message 3 of 7 , Apr 26, 2011
        • 0 Attachment
          Here is my understanding of the problem and suggestion

          Input is entered via B,
          A reads values and makes the values available to C
          C transmits values over the air to PC.

          I suggest that you need tests at system the whole end to end, module A, B, C and unit test level.

          You may build up the end to end tests through mocks and you will also need end to end test with the real system.

          Nader
          Sent from my BlackBerry� wireless device

          -----Original Message-----
          From: Steven Woody <narkewoody@...>
          Sender: extremeprogramming@yahoogroups.com
          Date: Tue, 26 Apr 2011 22:00:11
          To: <extremeprogramming@yahoogroups.com>
          Reply-To: extremeprogramming@yahoogroups.com
          Subject: [XP] Real or Unreal Test. Help me argue with my colleagues

          Hi,

          Our product is an embedded device. Looking externally, it is a
          powered box, say A. On one side of the box, their is a USB interface
          that need to be connected with another device B. Inside the box, there
          exists a wireless modem chip C. Our system, basically, is to read
          values from A, and make it readable via the wireless modem from a
          client program running on PC.

          User cannot directly play with the box A, he has to operate with
          device B and play with a PC read program on a PC.

          Now, we concern how the acceptance test for our box A should be done.
          My options is to mock device A and modem C, since I think the system
          under test is just A. But many colleagues think this kind of test is
          unreal and want to put the real device B and real modem C together in
          a test setup.

          My reasoning has not yet satisfying them so much. Would you please
          tell me your opinion and help me in argue with them?

          Thanks in advance.


          --
          Life is the only flaw in an otherwise perfect nonexistence
          � � -- Schopenhauer

          narke
          public key at http://subkeys.pgp.net:11371 (narkewoody@...)



          [Non-text portions of this message have been removed]
        • Steven Woody
          ... Thanks for the opinion. Maybe not to call it acceptance test, but system test? I am not so familiar to these definitions. The problem here is How to
          Message 4 of 7 , Apr 26, 2011
          • 0 Attachment
            On 26 April 2011 22:09, Curtis Cooley <curtis.cooley@...> wrote:

            >
            >
            > On Apr 26, 2011 7:00 AM, "Steven Woody" <narkewoody@...> wrote:
            > >
            > >
            > >
            > > Hi,
            > >
            > > Our product is an embedded device. Looking externally, it is a
            > > powered box, say A. On one side of the box, their is a USB interface
            > > that need to be connected with another device B. Inside the box, there
            > > exists a wireless modem chip C. Our system, basically, is to read
            > > values from A, and make it readable via the wireless modem from a
            > > client program running on PC.
            > >
            > > User cannot directly play with the box A, he has to operate with
            > > device B and play with a PC read program on a PC.
            > >
            > > Now, we concern how the acceptance test for our box A should be done.
            > > My options is to mock device A and modem C, since I think the system
            > > under test is just A. But many colleagues think this kind of test is
            > > unreal and want to put the real device B and real modem C together in
            > > a test setup.
            > >
            > > My reasoning has not yet satisfying them so much. Would you please
            > > tell me your opinion and help me in argue with them?
            > >
            >
            > In my opinion, acceptance testing should include as much of the 'real
            > world'
            > as practical. These are tests that don't necessarily have to run fast. They
            > are your regression suite that tells you when something is broken.
            >
            > A test/qa lab with real world equipment will provide the highest level of
            > confidence in the system.
            >
            >
            Thanks for the opinion. Maybe not to call it acceptance test, but system
            test? I am not so familiar to these definitions.

            The problem here is "How to control the real world in the terms of
            testing?". Making a value change to the device B can sometimes be a very
            hard job in our application, but to device A, we only need to ensure it can
            read a value back from B. If extern to the wireless mode, things can be
            even impossible, since there is no idea to control the GSM/GPRS network.



            > [Non-text portions of this message have been removed]
            >
            >
            >



            --
            Life is the only flaw in an otherwise perfect nonexistence
            -- Schopenhauer

            narke
            public key at http://subkeys.pgp.net:11371 (narkewoody@...)


            [Non-text portions of this message have been removed]
          • Steven Smith
            I think you re both right. In any system, you want to have a mixture of unit, integration, and acceptance tests. The exact mix of how many of these you d
            Message 5 of 7 , Apr 26, 2011
            • 0 Attachment
              I think you're both right. In any system, you want to have a mixture of
              unit, integration, and acceptance tests. The exact mix of how many of these
              you'd like to have (and how many you actually end up writing) will vary
              based on many factors, such as your team's experience with testing (and each
              kind of test) and the details of the application itself. Generally, you
              want to write as many tests as possible at the unit and integration end of
              the scale, as these tend to run faster and break less frequently than
              acceptance tests, which in many cases must be done manually in addition to
              having many more dependencies. I think the best way to approach the
              argument with your peers is to accept their side of the argument, but make
              the point that this kind of testing is a "defense in depth" approach, and
              there are things you could be catching with your unit and integration tests
              more easily and cheaply than with the acceptance tests on real hardware.

              Steve


              On Tue, Apr 26, 2011 at 10:00 AM, Steven Woody <narkewoody@...> wrote:

              >
              >
              > Hi,
              >
              > Our product is an embedded device. Looking externally, it is a
              > powered box, say A. On one side of the box, their is a USB interface
              > that need to be connected with another device B. Inside the box, there
              > exists a wireless modem chip C. Our system, basically, is to read
              > values from A, and make it readable via the wireless modem from a
              > client program running on PC.
              >
              > User cannot directly play with the box A, he has to operate with
              > device B and play with a PC read program on a PC.
              >
              > Now, we concern how the acceptance test for our box A should be done.
              > My options is to mock device A and modem C, since I think the system
              > under test is just A. But many colleagues think this kind of test is
              > unreal and want to put the real device B and real modem C together in
              > a test setup.
              >
              > My reasoning has not yet satisfying them so much. Would you please
              > tell me your opinion and help me in argue with them?
              >
              > Thanks in advance.
              >
              > --
              > Life is the only flaw in an otherwise perfect nonexistence
              > -- Schopenhauer
              >
              > narke
              > public key at http://subkeys.pgp.net:11371 (narkewoody@...)
              >
              >



              --
              Steve Smith
              http://SteveSmithBlog.com/
              http://twitter.com/ardalis


              [Non-text portions of this message have been removed]
            • Curtis Cooley
              ... This characterizes what I meant, but much more clearly. Thanks Steven. -- Curtis Cooley curtis.cooley@gmail.com home:http://curtiscooley.com
              Message 6 of 7 , Apr 26, 2011
              • 0 Attachment
                On Tue, Apr 26, 2011 at 7:38 AM, Steven Smith <ssmith.lists@...> wrote:
                > I think you're both right.  In any system, you want to have a mixture of
                > unit, integration, and acceptance tests.  The exact mix of how many of these
                > you'd like to have (and how many you actually end up writing) will vary
                > based on many factors, such as your team's experience with testing (and each
                > kind of test) and the details of the application itself.  Generally, you
                > want to write as many tests as possible at the unit and integration end of
                > the scale, as these tend to run faster and break less frequently than
                > acceptance tests, which in many cases must be done manually in addition to
                > having many more dependencies.  I think the best way to approach the
                > argument with your peers is to accept their side of the argument, but make
                > the point that this kind of testing is a "defense in depth" approach, and
                > there are things you could be catching with your unit and integration tests
                > more easily and cheaply than with the acceptance tests on real hardware.
                >

                This characterizes what I meant, but much more clearly. Thanks Steven.
                --
                Curtis Cooley
                curtis.cooley@...
                home:http://curtiscooley.com
                blog:http://ponderingobjectorienteddesign.blogspot.com
                ===============
                Leadership is a potent combination of strategy and character. But if
                you must be without one, be without the strategy.
                -- H. Norman Schwarzkopf
              • Steven Woody
                Thanks a lot!! I basically think same with you, but your language is so much better! On Apr 26, 2011 10:39 PM, Steven Smith wrote:
                Message 7 of 7 , Apr 26, 2011
                • 0 Attachment
                  Thanks a lot!! I basically think same with you, but your language is so much
                  better!
                  On Apr 26, 2011 10:39 PM, "Steven Smith" <ssmith.lists@...> wrote:


                  [Non-text portions of this message have been removed]
                Your message has been successfully submitted and would be delivered to recipients shortly.