Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Let's try to break things...

Expand Messages
  • Joi Ellis
    ... [snip] ... I suggest that you continue to harden your data entry routines. I know from personal experience how many times a developer s intuition and
    Message 1 of 9 , Aug 31, 2000
    • 0 Attachment
      Walter van Iterson wrote:
      >
      [snip]
      >
      > When I try to fix all these "breakable, but not likely to be
      > broken" parts everywhere in my program, I'll end up with having
      > three times as much code, of which two-thirds will probably
      > never be used. It makes the code more stable, though.
      >
      > Should I continue thinking "How can I break it", or shouldn't I?

      I suggest that you continue to harden your data entry routines.

      I know from personal experience how many times a developer's intuition and
      understanding of the cosmic all can be so radically different from my own.
      I've stumbled into every possible "it will never happen" trap they
      left unguarded in their application.

      I can speak of this from both the user *and* the developer side of
      things--my brain is wired strangely. I've learned many times that
      what makes perfect sense to me is incomprehensible to nearly everyone
      else, and vice versa.

      Put the checks in. Your crack-brained users will thank you.

      I can give you an extreme example of this... a product I use daily
      has a File/New menu option which leads to a pile of wizards for
      creating new instances of various complex and not-so-complex things.

      But, there isn't an option there for creating a new simple thing.
      For that, you use File/Open, and give it a name which does not yet
      exist. As far as I know, this is the only type of thing which must
      be created in this fashion.

      It makes perfect sense to the development team, and not a whit to
      me.

      (You be quiet, Eric!)

      --
      Joi Ellis Software Engineer
      Aravox Technologies joi@..., gyles19@...

      No matter what we think of Linux versus FreeBSD, etc., the one thing I
      really like about Linux is that it has Microsoft worried. Anything
      that kicks a monopoly in the pants has got to be good for something.
      - Chris Johnson
    • acockburn@aol.com
      Message 2 of 9 , Aug 31, 2000
      • 0 Attachment
        <<stepped up to the computer
        and slammed the keyboard with all of her fingers.
        The developer's eyes bugged out and then the
        app crashed.>>

        Happens to me all the time, with many/most applications, although I'm
        not so violent ... I'm just working on many things simultaneously and
        typing ahead of the program... with the same net effect
      • Scott Jackson
        hadda buddy who tested 14 drives for Sperry-Univac. Lotsa intentional thrashing for electro-mechanical stressing. As he walked out for a break, a happy
        Message 3 of 9 , Aug 31, 2000
        • 0 Attachment
          hadda buddy who tested 14" drives for Sperry-Univac.
          Lotsa intentional thrashing for electro-mechanical stressing.

          As he walked out for a break, a happy programmer said this sucka is
          booo leettt proof.

          As you described, my buddy grabbed a handful of keys and crashed it.
          the crestfallen programmer asked why'd you do that.
          the answer - 'users do it all the time" bada BOOM

          -----Original Message-----
          From: Michael C. Feathers [mailto:mfeathers@...]
          Sent: Thursday, August 31, 2000 8:57 AM
          To: extremeprogramming@egroups.com
          Subject: Re: [XP] Let's try to break things...



          ----- Original Message -----
          From: Joi Ellis <joi@...>
          > I suggest that you continue to harden your data entry routines.
          >
          > I know from personal experience how many times a
          > developer's intuition and understanding of the cosmic
          > all can be so radically different from my own. I've
          > stumbled into every possible "it will never happen" trap
          > they left unguarded in their application.

          Funny story on this one. I worked with a guy who
          was working on a routine to parse information
          from an edit box. He wanted an actual user
          to test it out so he called over a young statistician
          we'd just hired. He said "Rachel, please type in
          some bad input here and we'll see how my routine
          is working." Rachel stepped up to the computer
          and slammed the keyboard with all of her fingers.
          The developer's eyes bugged out and then the
          app crashed.


          ---------------------------------------------------
          Michael Feathers mfeathers@...
          Object Mentor Inc. www.objectmentor.com
          Training/Mentoring/Development
          -----------------------------------------------------
          "Only puny secrets need protection.
          Big discoveries are protected by
          public incredulity." --MarshallMcLuhan



          To Post a message, send it to: extremeprogramming@...

          To Unsubscribe, send a blank message to:
          extremeprogramming-unsubscribe@...

          Ad-free courtesy of objectmentor.com
        • Michael C. Feathers
          ... From: Joi Ellis ... Funny story on this one. I worked with a guy who was working on a routine to parse information from an edit box. He
          Message 4 of 9 , Aug 31, 2000
          • 0 Attachment
            ----- Original Message -----
            From: Joi Ellis <joi@...>
            > I suggest that you continue to harden your data entry routines.
            >
            > I know from personal experience how many times a
            > developer's intuition and understanding of the cosmic
            > all can be so radically different from my own. I've
            > stumbled into every possible "it will never happen" trap
            > they left unguarded in their application.

            Funny story on this one. I worked with a guy who
            was working on a routine to parse information
            from an edit box. He wanted an actual user
            to test it out so he called over a young statistician
            we'd just hired. He said "Rachel, please type in
            some bad input here and we'll see how my routine
            is working." Rachel stepped up to the computer
            and slammed the keyboard with all of her fingers.
            The developer's eyes bugged out and then the
            app crashed.


            ---------------------------------------------------
            Michael Feathers mfeathers@...
            Object Mentor Inc. www.objectmentor.com
            Training/Mentoring/Development
            -----------------------------------------------------
            "Only puny secrets need protection.
            Big discoveries are protected by
            public incredulity." --MarshallMcLuhan
          • Glew, Andy
            ... Yes. But... it s hard to think how can I break it for your own code. This is often cited as the reason for a separate validation group. The programmer
            Message 5 of 9 , Aug 31, 2000
            • 0 Attachment
              > Should I continue thinking "How can I break it", or shouldn't I?


              Yes. But... it's hard to think "how can I break it" for your
              own code. This is often cited as the reason for a separate
              validation group. The programmer has the "I want it to work"
              mindset, whereas the testers should have the "How could it
              break" mindset.

              But XP is good - taking responsibility for ensuring your
              code works is good. Writing unit tests often leads you to
              better code.

              So, code your own unit tests. Write tests first.

              But, if you have the opportunity to avail yourself of an
              "adversary" testing group with the "how can I break it"
              mentality, do so. Ask them to write tests for your code.
              (Automated, etc.) Remember, every test they write that
              breaks your code is one step towards making your code
              better.

              Nevertheless, if you don't have such a testing team,
              and even if you do, sometimes it's a good idea to try
              to put yourself into the "how can I break it" role.
              Ideally before you write the code, but sometimes afterwards.
              Especially after you have found one bug that wasn't caught
              by your "tests first" unit tests.

              Another useful trick is to trade off with one of your
              programming team buddies. Maybe while pairing, but
              maybe separately: pair1 wil try to test pair2's code,
              and vice versa. Actually, I've not done such trade-off
              testing with pairs, only on an individual basis, but it
              can help.

              ===

              An extended question is, can my code be made safe in
              all possible circumstances. Ideally yes, but sometimes no.

              Sometimes your code necessarily makes assumptions - you're
              passed a valid file descriptor, somebody else has checked
              that there is enough diskspace free (or reserved it, on
              proper OSes). Ideally you might have asserts for each
              of these assumptions, exercised only in debug mode, but
              sometimes you either cannot afford to have these assertions
              in real code, or you don't have access to the data needed
              to properly check the assertions.

              (I'm working with a module recently where the assertions
              are actually several times larger - more code, more parameters,
              etc. - than the actual code. The assertions require global
              knowledge that the module doesn't have. I hope to refactor
              this so that the module assumptions are more cleanly
              separated friom the global state, but that will take a long
              time. Legacy code. Owned by someone else, in another group,
              another building, another company.)

              With asserts in your code, you can write tests that fail cleanly,
              as expected - i.e. they pass. Be aware that the code may one
              day be extended to do something reasonable for such a situation,
              relaxing the constraints it places on the user - meaning that
              such an expected failure test may one day actually pass
              unexpectedly.

              Without asserts, well, you can't test for a "correct" result,
              there isn't necessarily a clean error mechanism, and checking all
              the possible forms of errors is infeasible. Some people say that
              if you give an undefined input to a function, any undefined
              behaviour is legitimate, ranging from crashing to rebooting the system
              to playing Hanoi, but most of us live in a more nuanced world.
              E.g. give a system an input for which its behaviour is not exactly
              specified, but you may not you may be able to say that it should
              not crash. You don't know what it should do, but it should not
              misbehave. Secure systems often work this way.

              And even if you can't write such negative tests, it may be worth
              keeping around such "known and expected to fail in funny ways"
              tests, especially if they were hard to write. One day you may
              enhance the behaviour.
            • cg@cdegroot.com
              ... Don t forget how can I break the tests role - make the code give wrong answers and see whether the tests catch it. Useful way to spend your time when you
              Message 6 of 9 , Aug 31, 2000
              • 0 Attachment
                Glew, Andy <andy.glew@...> said:
                >Nevertheless, if you don't have such a testing team,
                >and even if you do, sometimes it's a good idea to try
                >to put yourself into the "how can I break it" role.

                Don't forget "how can I break the tests" role - make the code give
                wrong answers and see whether the tests catch it. Useful way to
                spend your time when you can't find a pairing partner...

                --
                Cees de Groot http://www.cdegroot.com <cg@...>
                GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B
                Forge your CipherSaber and list it: http://www.xs4all.nl/~cg/ciphersaber/
              • Mark Wilden
                ... From: Glew, Andy ... I agree. I find that when I used to test code, I wasn t really in make it break mode, I was in am I happy
                Message 7 of 9 , Aug 31, 2000
                • 0 Attachment
                  ----- Original Message -----
                  From: "Glew, Andy" <andy.glew@...>
                  > Yes. But... it's hard to think "how can I break it" for your
                  > own code. This is often cited as the reason for a separate
                  > validation group. The programmer has the "I want it to work"
                  > mindset, whereas the testers should have the "How could it
                  > break" mindset.
                  >
                  > But XP is good - taking responsibility for ensuring your
                  > code works is good. Writing unit tests often leads you to
                  > better code.

                  I agree. I find that when I used to "test" code, I wasn't really in "make it
                  break" mode, I was in "am I happy with this?" mode.

                  But when I'm writing unit tests before I've formed any attachment to the
                  tested code (because I haven't written it yet!), I am very much in "make it
                  break" mode, and this is a benefit of XP.

                  XP is a collection of techniques that each reinforce the other. However,
                  many of its techniques are applicable by themselves to any techniquology,
                  and unit testing is perhaps the most important.
                • Riesbeck
                  ... Many years ago, in the era of 286 s and Apple II s, our main tester had a set of standard tests he applied to each program. My favorite was the elbow
                  Message 8 of 9 , Aug 31, 2000
                  • 0 Attachment
                    >----- Original Message -----
                    >
                    >Funny story on this one. I worked with a guy who
                    >was working on a routine to parse information
                    >from an edit box. He wanted an actual user
                    >to test it out so he called over a young statistician
                    >we'd just hired. He said "Rachel, please type in
                    >some bad input here and we'll see how my routine
                    >is working." Rachel stepped up to the computer
                    >and slammed the keyboard with all of her fingers.
                    >The developer's eyes bugged out and then the
                    >app crashed.

                    Many years ago, in the era of 286's and Apple II's, our main tester
                    had a set of standard tests he applied to each program. My favorite
                    was "the elbow test," where he ran his elbow over the keyboard for
                    half a minute.

                    Not surprisingly, he signed his reports "Doctor Death."

                    More surprisingly, once they knew this was coming, the programmers
                    actually developed applications that passed.



                    Chris Riesbeck
                    --------------------------------
                    http://www.cs.nwu.edu/~riesbeck/
                    --------------------------------
                  • Walter van Iterson
                    ... Did so, and ended up in how can I hack thinks -mode. Ultimately, it turned out in a school-example of refactoring. In detail: There s a class
                    Message 9 of 9 , Sep 1, 2000
                    • 0 Attachment
                      --- In extremeprogramming@egroups.com, Joi Ellis <joi@a...> wrote:
                      > [snip]
                      > >
                      > > Should I continue thinking "How can I break it", or shouldn't I?
                      >
                      > I suggest that you continue to harden your data entry routines.

                      Did so, and ended up in "how can I hack thinks"-mode. Ultimately,
                      it turned out in a school-example of refactoring.

                      In detail:

                      There's a class ImageAnalyzer, which has an image and stores
                      its width and height. Calculations on width and height should
                      never cause an ArrayIndexOutOfBoundsException.

                      There's a method
                      "public void setImageData(int[] imageData, int width, int height)"
                      which is supposed to throw an exception when an invalid width and
                      height might possibly lead to ArrayIndexOutOfBoundsExceptions
                      elsewhere in the code.


                      First attempt:
                      TEST: setImageData(new int[10 * 10], 10, 11);
                      SOLUTION: if (width * height != imageData.length) { exception }


                      Second attempt:
                      TEST: setImageData(int int[10 * 10], -10, -10);
                      SOLUTION: if (width < 0) { exception }


                      Third attempt: (that's where the hacker in me wakes up)
                      TEST: setImageData(new int[10 * 10], 116, 37025581);
                      // 116 * 37025581 equals (2^32)+100, causing an
                      // overflow with the value 100
                      SOLUTION: if ((width > imageData.length)
                      || (height > imageData.length)) { exception }

                      This overflow can probably also happen for a
                      multiplication of a positive and a negative value,
                      so change solution #2 to:

                      if ((width < 0) || (height < 0)) { exception }


                      Fourth attempt:
                      TEST: setImageData(new int[2000 * 2000], 59761, 71936);
                      // 59761 * 71936 equals (2^32)+4000000
                      SOLUTION (A): (yesterday) consider giving up catching everything
                      SOLUTION (B): (after a good night's sleep)
                      change solution #1 to
                      if ((long)width * height != imageData.length) { exc. }
                      remove solution #3
                      change solution #2 back to the original.

                      Now all hacking attempts are caught, even with less error checks
                      (3) than I originally feared (4 and it still doesn't catch
                      everything). Don't know if somebody after me will understand why
                      there's that (long) cast. Well, the test suite will tell them.

                      Or does anybody have yet another way to break it? ;-)

                      Walter
                    Your message has been successfully submitted and would be delivered to recipients shortly.