Loading ...
Sorry, an error occurred while loading the content.

RE: [XP] Test Organisation/Comments

Expand Messages
  • Philip Doherty
    Hi All, I m getting the developers in my company to experiment with TDD, the obvious concern from the managers is that the test cases aren t clearly visible to
    Message 1 of 13 , Feb 10, 2006
    • 0 Attachment
      Hi All,

      I'm getting the developers in my company to experiment with TDD, the
      obvious concern from the managers is that the test cases aren't clearly
      visible to them if they arent documented. How do you guys organise and
      comment your tests? And then how do you find information on these tests
      quickly, does anyone use documentation generation, I know documentation
      isn't favourable but if its automatically generated from the tests then
      I see no problem with this.

      Any ideas welcome.

      (Using VB.NET, Nunit)

      Philip Doherty
    • Ron Jeffries
      ... I d like to understand better what s behind this concern: it s not obvious to me. Why do the managers want the tests to be visible to them? What would that
      Message 2 of 13 , Feb 10, 2006
      • 0 Attachment
        On Friday, February 10, 2006, at 5:54:33 AM, Philip Doherty wrote:

        > I'm getting the developers in my company to experiment with TDD, the
        > obvious concern from the managers is that the test cases aren't clearly
        > visible to them if they arent documented.

        I'd like to understand better what's behind this concern: it's not
        obvious to me. Why do the managers want the tests to be visible to
        them? What would that give them? Do these managers read the code
        that the team is now writing?

        IMO, a really good XP team has customer tests (not programmer TDD
        tests) that are visible to customers and managers. These tests show
        that the system works as intended. The TDD tests are more a
        programmer thing. Thus my questions.

        > How do you guys organise and
        > comment your tests? And then how do you find information on these tests
        > quickly, does anyone use documentation generation, I know documentation
        > isn't favourable but if its automatically generated from the tests then
        > I see no problem with this.

        I keep all my tests in consistently-named classes related to the
        class they test or the feature they test. I don't print anything out
        because I don't need anything printed out. I'm wondering what you
        and your team and your managers would need documented, and what they
        think this documentation would give them.

        Regards,

        Ron Jeffries
        www.XProgramming.com
        Sorry about your cow ... I didn't know she was sacred.
      • Philip Doherty
        ... obvious to me. Why do the managers want the tests to be visible to them? What would that give them? Do these managers read the code that the team is now
        Message 3 of 13 , Feb 10, 2006
        • 0 Attachment
          >I'd like to understand better what's behind this concern: it's not
          obvious to me. Why do the managers want the tests to be visible to them?
          What would that give them? Do these managers read the code that the team
          is now >writing?
          >IMO, a really good XP team has customer tests (not programmer TDD
          >tests) that are visible to customers and managers. These tests show
          that the system works as intended. The TDD tests are more a programmer
          thing. Thus my questions.

          The automated tests are essentially the business rules and customer
          tests in one (and obviously a lot more), the managers need to see these
          if the tests are going to replace unit test plans which have to be
          updated as well as the tests. Where and how do you maintain and show
          your Customer tests?

          >I keep all my tests in consistently-named classes related to the class
          they test or the feature they test. I don't print anything out because I
          don't
          >need anything printed out. I'm wondering what you and your team and
          your managers would need documented, and what they think this
          documentation would give them.

          Why would you not need to see your tests? The documentation lets you see
          what you are testing, which is essentially your safety net regarding
          change. Sure you can look at a code coverage figure but that doesnt tell
          you what you are testing.

          Another valid reason is that the documentation (I stress automatically
          generated) would tick alot of quality standard boxs which is necessary
          for alot of our contracts.

          Regards

          Phil Doherty



          [Non-text portions of this message have been removed]
        • Doug Swartz
          ... Plus, there s a GUI which brings up the list of all the tests for anyone to see. (including a manager if she so desires) The GUI probably organizes the
          Message 4 of 13 , Feb 10, 2006
          • 0 Attachment
            Friday, February 10, 2006, 5:46:37 AM, Ron Jeffries wrote:

            > On Friday, February 10, 2006, at 5:54:33 AM, Philip Doherty wrote:

            >> I'm getting the developers in my company to experiment with TDD, the
            >> obvious concern from the managers is that the test cases aren't clearly
            >> visible to them if they arent documented.

            > I'd like to understand better what's behind this concern: it's not
            > obvious to me. Why do the managers want the tests to be visible to
            > them? What would that give them? Do these managers read the code
            > that the team is now writing?

            > IMO, a really good XP team has customer tests (not programmer TDD
            > tests) that are visible to customers and managers. These tests show
            > that the system works as intended. The TDD tests are more a
            > programmer thing. Thus my questions.

            >> How do you guys organise and
            >> comment your tests? And then how do you find information on these tests
            >> quickly, does anyone use documentation generation, I know documentation
            >> isn't favourable but if its automatically generated from the tests then
            >> I see no problem with this.

            > I keep all my tests in consistently-named classes related to the
            > class they test or the feature they test. I don't print anything out
            > because I don't need anything printed out. I'm wondering what you
            > and your team and your managers would need documented, and what they
            > think this documentation would give them.

            Plus, there's a GUI which brings up the list of all the tests
            for anyone to see. (including a manager if she so desires) The
            GUI probably organizes the tests into suites or categories of
            tests.

            The test runner may keep statistics, as well.

            --

            Doug Swartz
            daswartz@...
          • Philip Doherty
            ... anyone to see. (including a manager if she so desires) The GUI probably organizes the tests into suites or categories of tests. ... Hi Doug, What GUI are
            Message 5 of 13 , Feb 10, 2006
            • 0 Attachment
              Doug Swartz wrote:

              >Plus, there's a GUI which brings up the list of all the tests for
              anyone to see. (including a manager if she so desires) The GUI probably
              organizes the >tests into suites or categories of tests.

              >The test runner may keep statistics, as well.



              Hi Doug,

              What GUI are you talking about?


              Regards

              Phil
            • Doug Swartz
              ... is now writing? ... We maintain the tests in whatever form the test tool (Fitnesse, homegrown framework, etc.) requires. We use descriptive names for the
              Message 6 of 13 , Feb 10, 2006
              • 0 Attachment
                Friday, February 10, 2006, 6:13:25 AM, Philip Doherty wrote:

                > >I'd like to understand better what's behind this concern: it's not
                > obvious to me. Why do the managers want the tests to be visible to them?
                > What would that give them? Do these managers read the code that the team
                is now >>writing?
                >>IMO, a really good XP team has customer tests (not programmer TDD
                >>tests) that are visible to customers and managers. These tests show
                >> that the system works as intended. The TDD tests are more a programmer
                >> thing. Thus my questions.

                > The automated tests are essentially the business rules and customer
                > tests in one (and obviously a lot more), the managers need to see these
                > if the tests are going to replace unit test plans which have to be
                > updated as well as the tests. Where and how do you maintain and show
                > your Customer tests?

                We maintain the tests in whatever form the test tool
                (Fitnesse, homegrown framework, etc.) requires.
                We use descriptive names for the tests. The test tool prints
                out and saves results every time it's run. Anyone can see the
                latest report at any time.

                >>I keep all my tests in consistently-named classes related to the class
                > they test or the feature they test. I don't print anything out because I
                > don't
                >>need anything printed out. I'm wondering what you and your team and
                > your managers would need documented, and what they think this
                > documentation would give them.

                > Why would you not need to see your tests? The documentation lets you see
                > what you are testing, which is essentially your safety net regarding
                > change. Sure you can look at a code coverage figure but that doesnt tell
                > you what you are testing.

                Of course you need to see the tests. We're trying to
                understand what documentation beyond the tests themselves, and
                the results of the tests, your managers need to see.

                > Another valid reason is that the documentation (I stress automatically
                > generated) would tick alot of quality standard boxs which is necessary
                > for alot of our contracts.

                What kind of documentation do you use today for your tests to
                tick off these boxes? Is it something beyond a setup (or input
                data) scenario and description of the expected outputs?



                --

                Doug Swartz
                daswartz@...
              • Ron Jeffries
                ... Yes. Did you mean to send this to the list? Ron Jeffries www.XProgramming.com Don t confuse more precise with better. -- Brian Marick
                Message 7 of 13 , Feb 10, 2006
                • 0 Attachment
                  On Friday, February 10, 2006, at 8:02:12 AM, Doug Swartz wrote:

                  >> I keep all my tests in consistently-named classes related to the
                  >> class they test or the feature they test. I don't print anything out
                  >> because I don't need anything printed out. I'm wondering what you
                  >> and your team and your managers would need documented, and what they
                  >> think this documentation would give them.

                  > Plus, there's a GUI which brings up the list of all the tests
                  > for anyone to see. (including a manager if she so desires) The
                  > GUI probably organizes the tests into suites or categories of
                  > tests.

                  > The test runner may keep statistics, as well.

                  Yes. Did you mean to send this to the list?

                  Ron Jeffries
                  www.XProgramming.com
                  Don't confuse more precise with better. -- Brian Marick
                • Ron Jeffries
                  ... Oh ... you did. The address format confused me. My mistake. As is not uncommon. Ron Jeffries www.XProgramming.com You can observe a lot by watching. --Yogi
                  Message 8 of 13 , Feb 10, 2006
                  • 0 Attachment
                    On Friday, February 10, 2006, at 8:14:25 AM, Ron Jeffries wrote:

                    >> The test runner may keep statistics, as well.

                    > Yes. Did you mean to send this to the list?

                    Oh ... you did. The address format confused me. My mistake. As is
                    not uncommon.

                    Ron Jeffries
                    www.XProgramming.com
                    You can observe a lot by watching. --Yogi Berra
                  • Philip Doherty
                    ... homegrown framework, etc.) requires. ... saves results every time it s run. Anyone can see the latest report at any time. Ah, I only meant to discuss unit
                    Message 9 of 13 , Feb 10, 2006
                    • 0 Attachment
                      >We maintain the tests in whatever form the test tool (Fitnesse,
                      homegrown framework, etc.) requires.
                      >We use descriptive names for the tests. The test tool prints out and
                      saves results every time it's run. Anyone can see the latest report at
                      any time.

                      Ah, I only meant to discuss unit tests, unfortunately I havent started
                      with automated customer tests yet (FIT etc), its next on the list. As I
                      want to replace manual testing I think giving the managers the ability
                      to see what is being tested will make the transition smoother. Also
                      everyone (including myself) are starting this with a small amount of
                      automated testing experience and I think being able to view all of the
                      tests outside of a code editor would be a beneficial learning experience
                      for us all.

                      Am I correct in assuming that from your answers that you view customer
                      tests as more beneficial? I like the simplicity of them, especially
                      since a non IT customer should be creating them, but I also like the
                      detail which testing unit tests give me.


                      >Of course you need to see the tests. We're trying to understand what
                      documentation beyond the tests themselves, and the results of the tests,
                      your managers need to see.

                      I wanted to include a comment with the unit test and for this to be
                      extracted into the document for all to see. I suppose the comment would
                      be an explanation of the test and any business rule (or in the future,
                      customer test) it relates too. Nunit gives me the summary details of the
                      tests but I want the non technical explanation of them, not the results.



                      >What kind of documentation do you use today for your tests to tick off
                      these boxes? Is it something beyond a setup (or input
                      >data) scenario and description of the expected outputs?

                      Its simple input output and date tested spreadsheets.

                      So have you used anything like this? I think I will try Ndoc for a while
                      and see how it goes.


                      --

                      Doug Swartz
                      daswartz@...



                      To Post a message, send it to: extremeprogramming@...

                      To Unsubscribe, send a blank message to:
                      extremeprogramming-unsubscribe@...

                      ad-free courtesy of objectmentor.com
                      Yahoo! Groups Links
                    • Ron Jeffries
                      ... Unit test plans? I m a bear of very simple brain but I d rather have unit tests, written with the code, than plans. And I m really not seeing why managers
                      Message 10 of 13 , Feb 10, 2006
                      • 0 Attachment
                        On Friday, February 10, 2006, at 7:13:25 AM, Philip Doherty wrote:

                        >>I'd like to understand better what's behind this concern: it's not
                        >>obvious to me. Why do the managers want the tests to be visible to
                        >>them? What would that give them? Do these managers read the code
                        >>that the team is now writing?

                        >>IMO, a really good XP team has customer tests (not programmer TDD
                        >>tests) that are visible to customers and managers. These tests
                        >>show that the system works as intended. The TDD tests are more a
                        >>programmer thing. Thus my questions.

                        > The automated tests are essentially the business rules and customer
                        > tests in one (and obviously a lot more), the managers need to see these
                        > if the tests are going to replace unit test plans which have to be
                        > updated as well as the tests.

                        Unit test plans? I'm a bear of very simple brain but I'd rather have
                        unit tests, written with the code, than plans. And I'm really not
                        seeing why managers would be inspecting unit test plans. I must be
                        from another planet, which is often suggested anyway ...

                        > Where and how do you maintain and show your Customer tests?

                        Fit or Fitnesse are one good place. A custom-made customer test GUI
                        is used in some situations. Some teams just keep them in the hands
                        of testers or in books. I don't find that very satisfying.

                        >>I keep all my tests in consistently-named classes related to the
                        >>class they test or the feature they test. I don't print anything
                        >>out because I don't need anything printed out. I'm wondering what
                        >>you and your team and your managers would need documented, and
                        >>what they think this documentation would give them.

                        > Why would you not need to see your tests? The documentation lets you see
                        > what you are testing, which is essentially your safety net regarding
                        > change. Sure you can look at a code coverage figure but that doesnt tell
                        > you what you are testing.

                        Writing the tests, I test what I think needs testing. So I know what
                        I'm testing. You're in a situation where someone else wants to know
                        what you are testing. That seems odd to me ... so I must be missing
                        something.

                        The tests have names, and I can tell from the names what they test.
                        If I want more detail, I can look at the tests. What good is a typed
                        copy that may or may not match the reality? To the programmers, I'd
                        suggest "not much".

                        > Another valid reason is that the documentation (I stress automatically
                        > generated) would tick alot of quality standard boxs which is necessary
                        > for alot of our contracts.

                        Well, by all means, if you want it, or need it, do it. But in my
                        model of how things ought to be done, the managers and customers
                        should specify their own tests if they want tests, because unit
                        tests work best when they are at a level of fineness that won't tell
                        managers and customers much.

                        I'm getting a vibe of not much trust in your organization. That's
                        often the way things start when teams start to go Agile, because up
                        until now, managers and customers haven't had much to watch. When a
                        team starts to deliver running tested features, that can change
                        rapidly. Until then, of course, we have to do whatever we must do to
                        keep trust in place. Even if it's odd.

                        Ron Jeffries
                        www.XProgramming.com
                        In times of stress, I like to turn to the wisdom of my Portuguese waitress,
                        who said: "Olá, meu nome é Marisol e eu serei sua garçonete."
                        -- after Mark Vaughn, Autoweek.
                      • Charlie Poole
                        Assuming you also have automated customer/acceptance tests, your management should really not be concerned about the content of the unit tests. Depending on
                        Message 11 of 13 , Feb 10, 2006
                        • 0 Attachment
                          Assuming you also have automated customer/acceptance tests, your management
                          should really not be concerned about the content of the unit tests.
                          Depending on their level of experience with a TDD approach - I'm guessing
                          none - they may want to monitor things like the amount of testing you are
                          doing, whether all tests are passing, and maybe the test coverage*

                          That's OK because - except for the last - it's info you need anyway and
                          which is produced for you by the framework.

                          If you don't have automated acceptance tests, I think you'd be better off
                          spending your time creating them than indulging in the exercise you've
                          described. With a framework like FIT, you can put all the explanation
                          anybody might ever want right in the html pages that embody the tests.

                          Charlie

                          *Test coverage is not code coverage - it's simply some sort of report on how
                          well tested each part of the application itself is. I usually generate such
                          reports from the test assembly metadata, without using the source code. If
                          you really wanted to include doc comments, I'd put them on the production
                          classes, not on the tests. So your report would say something like...

                          Class: XyzClass Summary: xxxxx xxxxxxx xxxxx (from your doc comments)
                          5 public methods, 8 tests

                          This isn't rocket science, but it does take time to create and maintain. Try
                          to go with just the testing volume and pass rate first. Try to do it without
                          the doc comments and just name your classes so they make sense.

                          > -----Original Message-----
                          > From: extremeprogramming@yahoogroups.com
                          > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of
                          > Philip Doherty
                          > Sent: Friday, February 10, 2006 4:13 AM
                          > To: extremeprogramming@yahoogroups.com
                          > Subject: RE: [XP] Test Organisation/Comments
                          >
                          >
                          >
                          >
                          > >I'd like to understand better what's behind this concern:
                          > it's not obvious to me. Why do the managers want the tests to
                          > be visible to them?
                          > What would that give them? Do these managers read the code
                          > that the team is now >writing?
                          > >IMO, a really good XP team has customer tests (not programmer TDD
                          > >tests) that are visible to customers and managers. These tests show
                          > that the system works as intended. The TDD tests are more a
                          > programmer thing. Thus my questions.
                          >
                          > The automated tests are essentially the business rules and
                          > customer tests in one (and obviously a lot more), the
                          > managers need to see these if the tests are going to replace
                          > unit test plans which have to be updated as well as the
                          > tests. Where and how do you maintain and show your Customer tests?
                          >
                          > >I keep all my tests in consistently-named classes related to
                          > the class
                          > they test or the feature they test. I don't print anything
                          > out because I don't
                          > >need anything printed out. I'm wondering what you and your team and
                          > your managers would need documented, and what they think this
                          > documentation would give them.
                          >
                          > Why would you not need to see your tests? The documentation
                          > lets you see what you are testing, which is essentially your
                          > safety net regarding change. Sure you can look at a code
                          > coverage figure but that doesnt tell you what you are testing.
                          >
                          > Another valid reason is that the documentation (I stress automatically
                          > generated) would tick alot of quality standard boxs which is
                          > necessary for alot of our contracts.
                          >
                          > Regards
                          >
                          > Phil Doherty
                          >
                          >
                          >
                          > [Non-text portions of this message have been removed]
                          >
                          >
                          >
                          >
                          >
                          > To Post a message, send it to: extremeprogramming@...
                          >
                          > To Unsubscribe, send a blank message to:
                          > extremeprogramming-unsubscribe@...
                          >
                          > ad-free courtesy of objectmentor.com
                          > Yahoo! Groups Links
                          >
                          >
                          >
                          >
                          >
                          >
                          >
                        • Doug Swartz
                          ... Oh, then I misunderstood. I thought you referenced customer tests. I think learning to do automated tests, and especially, test driven development is a
                          Message 12 of 13 , Feb 10, 2006
                          • 0 Attachment
                            Friday, February 10, 2006, 7:43:29 AM, Philip Doherty wrote:


                            >>We maintain the tests in whatever form the test tool (Fitnesse,
                            > homegrown framework, etc.) requires.
                            >>We use descriptive names for the tests. The test tool prints out and
                            > saves results every time it's run. Anyone can see the latest report at
                            > any time.

                            > Ah, I only meant to discuss unit tests, unfortunately I havent started
                            > with automated customer tests yet (FIT etc), its next on the list. As I
                            > want to replace manual testing I think giving the managers the ability
                            > to see what is being tested will make the transition smoother. Also
                            > everyone (including myself) are starting this with a small amount of
                            > automated testing experience and I think being able to view all of the
                            > tests outside of a code editor would be a beneficial learning experience
                            > for us all.

                            Oh, then I misunderstood. I thought you referenced customer
                            tests.

                            I think learning to do automated tests, and especially, test
                            driven development is a crucial development skill. Personally,
                            though, I'd concentrate on creating good names for my tests
                            and test methods, rather than investing time in descriptive
                            comments. Programmer/unit tests really are mostly for the
                            developers. I haven't used Nunit, but our unit test runners
                            can also print out results. Again, I've never felt a
                            particular need for much more than a list of tests and the red
                            or green pop-up.

                            While code coverage metrics are only one portion of the
                            picture, they can be useful in showing management that your
                            new automated tests are exercising more and more of the
                            system. They are also pretty easy to create.

                            Wouldn't lists of tests (with descriptive names) and code
                            coverage metrics be a lot more than the managers have today
                            with manual testing?

                            > Am I correct in assuming that from your answers that you view customer
                            > tests as more beneficial? I like the simplicity of them, especially
                            > since a non IT customer should be creating them, but I also like the
                            > detail which testing unit tests give me.

                            Absolutely not! I consider both types to be of great value.

                            Programmers use programmer tests every day, all the time, to
                            help them do their work. They help us design our code as we're
                            writing it, and they give us a great safety net to ease our
                            fear of breaking something accidentally. They are our
                            microscope into the system.

                            Customer tests exercise and examine the system from a
                            different perspective. They give the customers, and the
                            developers, and the rest of the team, confidence that the
                            system works end-to-end and does, and continues to do, what
                            the customer needs. They are a macroscope on the system.

                            >>Of course you need to see the tests. We're trying to understand what
                            > documentation beyond the tests themselves, and the results of the tests,
                            > your managers need to see.

                            > I wanted to include a comment with the unit test and for this to be
                            > extracted into the document for all to see. I suppose the comment would
                            > be an explanation of the test and any business rule (or in the future,
                            > customer test) it relates too. Nunit gives me the summary details of the
                            > tests but I want the non technical explanation of them, not the results.

                            I imagine this could be useful, but I wouldn't do it unless
                            someone has already told me it is required. The vast majority
                            of the benefit of unit tests is for the programmers. the
                            programmers use the IDE. The unit test tools are already
                            integrated with the IDE. The additional documentation work
                            doesn't feel like much of a value add to me.

                            Good luck! You've started down a great path. Continue to ask
                            questions and experiment, but mostly, just keep writing tests!

                            --

                            Doug Swartz
                            daswartz@...
                          • Kent Beck
                            Philip, It sounds like your managers want to understand how well you are testing. If I was a manager, I would want to know this about my team. I find it
                            Message 13 of 13 , Feb 13, 2006
                            • 0 Attachment
                              Philip,

                              It sounds like your managers want to understand how well you are testing. If
                              I was a manager, I would want to know this about my team. I find it helpful
                              to look at my tests from a high level. An open-source tool like Emma or a
                              for-pay tool like Agitar's Management Dashboard [full disclosure: I work for
                              Agitar part-time] helps me understand where my tests are strong and where
                              they are weak. I particularly like the Dashboard because it rolls the
                              information up so managers can get whatever level of information they want.

                              Sincerely yours,

                              Kent Beck
                              Three Rivers Institute

                              > -----Original Message-----
                              > From: extremeprogramming@yahoogroups.com
                              > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of
                              > Philip Doherty
                              > Sent: Friday, February 10, 2006 2:55 AM
                              > To: extremeprogramming@yahoogroups.com
                              > Subject: RE: [XP] Test Organisation/Comments
                              >
                              >
                              > Hi All,
                              >
                              > I'm getting the developers in my company to experiment with TDD, the
                              > obvious concern from the managers is that the test cases
                              > aren't clearly
                              > visible to them if they arent documented. How do you guys organise and
                              > comment your tests? And then how do you find information on
                              > these tests
                              > quickly, does anyone use documentation generation, I know
                              > documentation
                              > isn't favourable but if its automatically generated from the
                              > tests then
                              > I see no problem with this.
                              >
                              > Any ideas welcome.
                              >
                              > (Using VB.NET, Nunit)
                              >
                              > Philip Doherty
                              >
                              >
                              > To Post a message, send it to: extremeprogramming@...
                              >
                              > To Unsubscribe, send a blank message to:
                              > extremeprogramming-unsubscribe@...
                              >
                              > ad-free courtesy of objectmentor.com
                              > Yahoo! Groups Links
                              >
                              >
                              >
                              >
                              >
                              >
                            Your message has been successfully submitted and would be delivered to recipients shortly.