Loading ...
Sorry, an error occurred while loading the content.

Requirement Traceability Matrix and Agile Testing

Expand Messages
  • Jorge Argus
    Since I m still fairly new to agile - I was just asked about adding a traceability matrix. Has anyone used a traceability matrix with agile testing? My first
    Message 1 of 16 , Mar 14, 2008
    View Source
    • 0 Attachment
      Since I'm still fairly new to agile - I was just asked about adding a
      traceability matrix. Has anyone used a traceability matrix with agile
      testing? My first response was that the team didn't need this - since
      it's built into our (company mangled) FIT, where everything is being
      covered. Has anyone else faced this question? Does anyone use a secret
      RTM on the side? Thanks.
    • Manuel MOLINIER
      Hi Jorge, if it is kind of there in FIT then you can probably very easily extract it in an easy to read Matrix. How I usually do is that I put on one Axis the
      Message 2 of 16 , Mar 14, 2008
      View Source
      • 0 Attachment
        Hi Jorge,
        if it is kind of there in FIT then you can probably very easily extract it in an easy to read Matrix.
        How I usually do is that I put on one Axis the requirements or user stories and on the other the test cases.
        I then fill the matrix so that in one glance you can track with test cases need to be modified if a requirement is changed and which test case to execute for this.
        It also allow to verify the requirements coverage at a glance.
        I'm not sure that requirements and tests in FIT has the same level of clarity/quick glance info providing.
         
        my 2 cents of it.
         
        Regards,
         
        Manuel


         
        On 3/14/08, Jorge Argus <arterberry@...> wrote:

        Since I'm still fairly new to agile - I was just asked about adding a
        traceability matrix. Has anyone used a traceability matrix with agile
        testing? My first response was that the team didn't need this - since
        it's built into our (company mangled) FIT, where everything is being
        covered. Has anyone else faced this question? Does anyone use a secret
        RTM on the side? Thanks.


      • Michael Bolton
        ... traceability matrix. Has anyone used a traceability matrix with agile testing? My first response was that the team didn t need this - since it s built into
        Message 3 of 16 , Mar 14, 2008
        View Source
        • 0 Attachment
          >Since I'm still fairly new to agile - I was just asked about adding a
          traceability matrix. Has anyone used a traceability matrix with agile
          testing? My first response was that the team didn't need this - since
          it's built into our (company mangled) FIT, where everything is being
          covered. Has anyone else faced this question? Does anyone use a secret
          RTM on the side? Thanks.

          It might be worthwhile to ask the people who are asking for a traceability
          matrix about the problem they perceive and are trying to solve. What
          alternatives are available to solve that problem?

          Also: what does it mean for /everything/ to be covered in FIT if the
          information that they seek is missing?

          ---Michael B.
        • Bradley, Todd
          If you re talking about a traceability matrix as being a document that shows connections between requirements and tests, I don t currently have one. But I d
          Message 4 of 16 , Mar 14, 2008
          View Source
          • 0 Attachment
            If you're talking about a traceability matrix as being a document that
            shows connections between requirements and tests, I don't currently have
            one. But I'd like to put one together. We develop tests in each
            sprint, as functionality is being developed. But requirements for a
            feature sometimes change in unpredictable ways later. So I'm always a
            little nervous that the test I ran 6 months ago doesn't really fully
            cover the requirement as it exists today.


            Todd.


            > -----Original Message-----
            > From: agile-testing@yahoogroups.com
            > [mailto:agile-testing@yahoogroups.com] On Behalf Of Jorge Argus
            > Sent: Friday, March 14, 2008 8:26 AM
            > To: agile-testing@yahoogroups.com
            > Subject: [agile-testing] Requirement Traceability Matrix and
            > Agile Testing
            >
            > Since I'm still fairly new to agile - I was just asked about
            > adding a traceability matrix. Has anyone used a traceability
            > matrix with agile testing? My first response was that the
            > team didn't need this - since it's built into our (company
            > mangled) FIT, where everything is being covered. Has anyone
            > else faced this question? Does anyone use a secret RTM on the
            > side? Thanks.
            >
            >
            > ------------------------------------
            >
            > Yahoo! Groups Links
            >
            >
            >
            >
            >
          • Ron Jeffries
            ... What prevents you from writing new tests when requirements change? Ron Jeffries www.XProgramming.com Computers are useless. They can only give you answers.
            Message 5 of 16 , Mar 14, 2008
            View Source
            • 0 Attachment
              Hello, Todd. On Friday, March 14, 2008, at 1:55:20 PM, you wrote:

              > But requirements for a
              > feature sometimes change in unpredictable ways later. So I'm always a
              > little nervous that the test I ran 6 months ago doesn't really fully
              > cover the requirement as it exists today.

              What prevents you from writing new tests when requirements change?

              Ron Jeffries
              www.XProgramming.com
              Computers are useless. They can only give you answers. -- Picasso
            • Bradley, Todd
              ... If you don t have some sort of link between requirements and tests, how do you know which tests are dirty and need to be updated due to a requirements
              Message 6 of 16 , Mar 14, 2008
              View Source
              • 0 Attachment
                > Hello, Todd. On Friday, March 14, 2008, at 1:55:20 PM, you wrote:
                >
                > > But requirements for a
                > > feature sometimes change in unpredictable ways later. So
                > I'm always a
                > > little nervous that the test I ran 6 months ago doesn't
                > really fully
                > > cover the requirement as it exists today.
                >
                > What prevents you from writing new tests when requirements change?

                If you don't have some sort of link between requirements and tests, how
                do you know which tests are "dirty" and need to be updated due to a
                requirements change? That link is traceability.


                Todd.
              • Mike Emeigh
                Michael Bolton wrote: (snip) ... This is a classic time, in my experience, to apply the 5 Whys technique: http://en.wikipedia.org/wiki/5_Whys to find out the
                Message 7 of 16 , Mar 14, 2008
                View Source
                • 0 Attachment
                  Michael Bolton wrote:
                  (snip)
                  >
                  > It might be worthwhile to ask the people who are asking for a traceability
                  > matrix about the problem they perceive and are trying to solve. What
                  > alternatives are available to solve that problem?
                  >
                  > Also: what does it mean for /everything/ to be covered in FIT if the
                  > information that they seek is missing?

                  This is a classic time, in my experience, to apply the 5 Whys technique:

                  http://en.wikipedia.org/wiki/5_Whys

                  to find out the REAL issue. RTMs, in my experience, are almost never useful enough to justify the cost of maintaining them, even in a waterfall environment. The only reason I see for doing one is to satisfy some contract requirement, and even then I'd argue that the FIT suite would suffice to meet the requirement.
                  --
                  Mike Emeigh
                  piratefan1@...

                  "Quantification automatically privileges people who are fond of numbers and
                  marginalizes people who are math phobic." -- James Bach
                • Michael Bolton
                  Todd: But requirements for a feature sometimes change in unpredictable ways later. So ... really fully ... Ron: What prevents you from writing new tests
                  Message 8 of 16 , Mar 14, 2008
                  View Source
                  • 0 Attachment
                    Todd:>> But requirements for a feature sometimes change in unpredictable
                    ways later. So
                    >> I'm always a little nervous that the test I ran 6 months ago doesn't
                    really fully
                    >> cover the requirement as it exists today.

                    Ron:>> What prevents you from writing new tests when requirements change?

                    >If you don't have some sort of link between requirements and tests, how
                    do you know which tests are "dirty" and need to be updated due to a
                    requirements change? That link is traceability.

                    What's the difference between "traceability" and "a traceability document"?

                    ---Michael B.
                  • Bradley, Todd
                    ... The first is a data structure, and the second is a human-readable version of the data structure. Or maybe I don t understand the point of your question.
                    Message 9 of 16 , Mar 14, 2008
                    View Source
                    • 0 Attachment
                      Michael wrote:
                      > Todd:>> But requirements for a feature sometimes change in
                      > unpredictable ways later. So
                      > >> I'm always a little nervous that the test I ran 6 months
                      > ago doesn't
                      > really fully
                      > >> cover the requirement as it exists today.
                      >
                      > Ron:>> What prevents you from writing new tests when
                      > requirements change?
                      >
                      > >If you don't have some sort of link between requirements and
                      > tests, how
                      > do you know which tests are "dirty" and need to be updated
                      > due to a requirements change? That link is traceability.
                      >
                      > What's the difference between "traceability" and "a
                      > traceability document"?

                      The first is a data structure, and the second is a human-readable
                      version of the data structure. Or maybe I don't understand the point of
                      your question. Is there a point you're trying to make?


                      Todd.
                    • Michael Bolton
                      Michael: What s the difference between traceability and a traceability document ? Todd: The first is a data structure, and the second is a human-readable
                      Message 10 of 16 , Mar 14, 2008
                      View Source
                      • 0 Attachment
                        Michael:>> What's the difference between "traceability" and "a traceability
                        document"?

                        Todd:>The first is a data structure, and the second is a human-readable
                        version of the data structure. Or maybe I don't understand the point of
                        your question. Is there a point you're trying to make?

                        That's an interesting answer. I don't think of traceability as a data
                        structure as such. I think of it as something more abstract--a concept that
                        helps us understand the historical links between one set of ideas and
                        another, or an answer to a question in the general form of "Why did we do
                        that?" More extremely, it might be "Who did X, and what did they do, and
                        why, or when, or how did they do it?" Answers to those questions can take
                        many forms, depending on the culture. Conversations, stories, strategic
                        themes, histories, logs, journals, source code, automated tests, design
                        documents, daily scrums, email, and (yes) spreadsheets are all media in
                        which traceability might be found, and all of them can be passed from one
                        person to another. But only some of these forms are documents that are
                        explicitly designed for that purpose and for no other. I'd suggest that you
                        need traceability if there's a danger of losing corporate memory that is
                        genuinely valuable--but what's the risk of that when it's already available
                        in so many forms?

                        Now, there are plausibly legitimate answers to a question like this. The
                        trouble (and clutter, and extra expense, and extra overhead) tends to come
                        when we haven't asked the question at all. That would be the suggestion
                        that I would make to the Original Poster: Someone wants traceability? Ask
                        what they want it for, how often they'll want it, the form in which they'll
                        want it, the forms in which they might be able to accept it--and then ask if
                        the cost will be matched by value. Things that can be read by a human
                        typically require a human to write them--yet most projects I've seen tend to
                        be short on extra humans.

                        Recently I saw Alton Brown, the celebrity cook, on TV. He demonstrated a
                        gizmo that's designed to take the smell of garlic off your hands. It's
                        essentially a lump of stainless steel, and there's something about the
                        chemistry that removes the garlic. Cool idea. But he chucked it, saying
                        that you can get exactly the same effect by rubbing your hands on the flat
                        of your stainless steel chef's knife (or if you have one of those knives
                        with a stainless steel handle, you can use that). Why buy an extra
                        tchotchke to clutter up the kitchen when you've already got what you need?

                        Another variation on Mike Emeigh's good suggestion of the Five Whys, by the
                        way, is a set of questions that I first heard from Don Gray: 1) What will
                        happen if I do? 2) What won't happen if I do? 3) What will happen if I
                        don't? 4) What won't happen if I don't?

                        ---Michael B.
                      • Ron Jeffries
                        ... What prevents you from running the tests and seeing which ones break? What prevents you from naming the tests after the corresponding requirements stories?
                        Message 11 of 16 , Mar 14, 2008
                        View Source
                        • 0 Attachment
                          Hello, Todd. On Friday, March 14, 2008, at 3:26:52 PM, you wrote:

                          >> Hello, Todd. On Friday, March 14, 2008, at 1:55:20 PM, you wrote:
                          >>
                          >> > But requirements for a
                          >> > feature sometimes change in unpredictable ways later. So
                          >> I'm always a
                          >> > little nervous that the test I ran 6 months ago doesn't
                          >> really fully
                          >> > cover the requirement as it exists today.
                          >>
                          >> What prevents you from writing new tests when requirements change?

                          > If you don't have some sort of link between requirements and tests, how
                          > do you know which tests are "dirty" and need to be updated due to a
                          > requirements change? That link is traceability.

                          What prevents you from running the tests and seeing which ones
                          break?

                          What prevents you from naming the tests after the corresponding
                          requirements stories?

                          Ron Jeffries
                          www.XProgramming.com
                          If not now, when? -- The Talmud
                        • Doug Swartz
                          ... This is essentially what we do. We write new test scenarios to verify that the new requirement is working correctly. Then we run the pre-existing tests.
                          Message 12 of 16 , Mar 14, 2008
                          View Source
                          • 0 Attachment
                            Friday, March 14, 2008, 4:02:06 PM, Ron Jeffries wrote:

                            > Hello, Todd. On Friday, March 14, 2008, at 3:26:52 PM, you wrote:

                            >>> Hello, Todd. On Friday, March 14, 2008, at 1:55:20 PM, you wrote:
                            >>>
                            >>> > But requirements for a
                            >>> > feature sometimes change in unpredictable ways later. So
                            >>> I'm always a
                            >>> > little nervous that the test I ran 6 months ago doesn't
                            >>> really fully
                            >>> > cover the requirement as it exists today.
                            >>>
                            >>> What prevents you from writing new tests when requirements change?

                            >> If you don't have some sort of link between requirements and tests, how
                            >> do you know which tests are "dirty" and need to be updated due to a
                            >> requirements change? That link is traceability.

                            > What prevents you from running the tests and seeing which ones
                            > break?

                            This is essentially what we do. We write new test scenarios to
                            verify that the new requirement is working correctly. Then we
                            run the pre-existing tests. When any of them don't pass, we
                            examine the failure to see if we broke something we shouldn't
                            have; or if the old test needs to be updated to reflect the
                            changed requirements.

                            Perhaps it seems too simple to work, but it has, so far.


                            --

                            Doug Swartz
                            daswartz@...
                          • John Overbaugh
                            In the spirit of being agile... Who writes your RTVM and how is it/are they stored? Seems to me one of the liberating benefits of Agile is a reduction of
                            Message 13 of 16 , Mar 14, 2008
                            View Source
                            • 0 Attachment
                              In the spirit of being agile... Who writes your RTVM and how is it/are they stored? Seems to me one of the liberating benefits of Agile is a reduction of documentation - wherever it results in improved quality/decreased effort. In my role as TM for a major retailer's merchant system transformation project, we wasted months of time 1) reading technical and functional specs, 2) entering RTVM into one system, 3) exporting RTVM and importing into Quality Center and 4) mapping test cases.

                              Why can't the RTVM be THE test cases, or at least a sub-set of the cases? Why can't RTVM be entered as part of a user story (for instance, as acceptance tests or the likes)? Why can't test cases plus a deep test case review replace RTVM entirely?

                              Our system was imposed by years of practice as well as the presence of a certain, uh, blue partner. I was making progress - we did move to eliminate more than one of the four steps above, but habit is tough to break and it took a lot of time. I see the 'value' in that RTVM allows mapping of independent validation cases back to functional requirements. I think, though, that the whole concept of RTVM screams distrust of the testing organization. Having seen many test orgs, I can say that there are situations where that distrust is valid. There are, however, many more situations where the test org is just as committed and intrested in a quality release as the business is--in those circumstances, trusting the test org is one of the best things that can be done.

                              Oops - sorry, I got up on my soap box for a bit there!

                              Anyhow, there must be a way to eliminate the silos and just strive for efficient, effective test coverage.

                              John O.
                              http://thoughtsonqa.blogspot.com

                              On Fri, Mar 14, 2008 at 11:55 AM, Bradley, Todd <todd.bradley@...> wrote:

                              If you're talking about a traceability matrix as being a document that
                              shows connections between requirements and tests, I don't currently have
                              one. But I'd like to put one together. We develop tests in each
                              sprint, as functionality is being developed. But requirements for a
                              feature sometimes change in unpredictable ways later. So I'm always a
                              little nervous that the test I ran 6 months ago doesn't really fully
                              cover the requirement as it exists today.

                              Todd.



                              > -----Original Message-----
                              > From: agile-testing@yahoogroups.com
                              > [mailto:agile-testing@yahoogroups.com] On Behalf Of Jorge Argus
                              > Sent: Friday, March 14, 2008 8:26 AM
                              > To: agile-testing@yahoogroups.com
                              > Subject: [agile-testing] Requirement Traceability Matrix and
                              > Agile Testing
                              >
                              > Since I'm still fairly new to agile - I was just asked about
                              > adding a traceability matrix. Has anyone used a traceability
                              > matrix with agile testing? My first response was that the
                              > team didn't need this - since it's built into our (company
                              > mangled) FIT, where everything is being covered. Has anyone
                              > else faced this question? Does anyone use a secret RTM on the
                              > side? Thanks.
                              >
                              >
                              > ------------------------------------
                              >
                              > Yahoo! Groups Links
                              >
                              >
                              >
                              >
                              >



                              --
                              John Overbaugh
                              blog: http://thougthsonqa.blogspot.com
                              site: http://www.yourtestmanager.com
                              <<john at your test manager dot com>>
                            • Brad Appleton
                              ... Right on Michael! The difference between traceability and a traceability *document* or *matrix* is ... Traceability is a desired property of a project,
                              Message 14 of 16 , Mar 15, 2008
                              View Source
                              • 0 Attachment
                                Michael Bolton wrote:
                                > That's an interesting answer. I don't think of traceability as a data
                                > structure as such. I think of it as something more abstract--a concept that
                                > helps us understand the historical links between one set of ideas and
                                > another, or an answer to a question in the general form of "Why did we do
                                > that?"

                                Right on Michael!

                                The difference between "traceability" and a traceability *document* or
                                *matrix* is ...

                                "Traceability" is a desired property of a project, one that makes a
                                statement about the transparency and visibility of the "important"
                                pieces of information & artifacts and the ability to see and navigate
                                their interrelationships.

                                A traceability "document" or "matrix" is one presumed implementation of
                                providing this "desired property" (and a traceability matrix is not a
                                particularly agile way of doing it, assuming it has to be manually
                                created & maintained).

                                Manually created & updated/maintained traceability matrices and
                                documents are one of the most tedious, time-consuming, and tiresome ways
                                of "demonstrating" traceability. You want to find an easier way if at
                                all possible, one which is more or less automatics, or better yet, is
                                simply transparent as a result of how you do your work. Hence my mantra:

                                "Trustworthy Transparency /over/ Tiresome Traceability"

                                The bottom-line is that the extent of traceability you genuinely require
                                may be different from that which many might automatically assume. And
                                more importantly, for whatever degree of traceability you DO implement,
                                you want the corresponding traceability document/report to be something
                                comes almost automatically as a by-product of the transparency of your
                                process, or that you can at least auto-generate without a lot of effort
                                to update/maintain or to initially create.

                                For a more thorough "dissection" of what the presumed purpose and value
                                of traceability is, I recommend the article:
                                * "The Trouble with Tracing: Traceability Dissected"
                                <<http://www.cmcrossroads.com/content/view/6685/202/>>

                                To read about some lean/agile approaches to attaining traceability, take
                                a peek at:
                                * "Lean Traceability: a smattering of strategies and solutions"
                                <<http://www.cmcrossroads.com/content/view/9089/120>>

                                For what I consider the XP "take" on traceability, see:
                                * "eXtreme Traceability"
                                <<http://blog.bradapp.net/2006/04/extreme-traceability.html>>

                                (Other "Agile SCM" articles are at http://cmwiki.com/AgileSCMArticles)
                                --
                                Brad Appleton <brad {AT} bradapp.net>
                                Agile CM Environments (http://blog.bradapp.net/)
                                & Software CM Patterns (www.scmpatterns.com)
                                "And miles to go before I sleep" -- Robert Frost
                              • Bradley, Todd
                                ... to ... Due to a sad lack of automated test tool support for our chosen GUI technology, nearly all our tests are manual. So, I guess the answer to your
                                Message 15 of 16 , Mar 17, 2008
                                View Source
                                • 0 Attachment
                                  Ron Jeffries wrote:
                                  > > If you don't have some sort of link between requirements and tests,
                                  > > how do you know which tests are "dirty" and need to be updated due
                                  to
                                  > > a requirements change? That link is traceability.
                                  >
                                  > What prevents you from running the tests and seeing which ones break?

                                  Due to a sad lack of automated test tool support for our chosen
                                  GUI technology, nearly all our tests are manual. So, I guess the
                                  answer to your question is "time".

                                  > What prevents you from naming the tests after the
                                  > corresponding requirements stories?

                                  That's all well and good until someone comes along and renames
                                  the requirement. There's got to be a better link between the
                                  requirements and the tests than just having the same name.


                                  Michael Bolton wrote:
                                  > Michael:>> What's the difference between "traceability" and
                                  > "a traceability document"?
                                  >
                                  > Todd:>The first is a data structure, and the second is a
                                  > human-readable version of the data structure. Or maybe I
                                  > don't understand the point of your question. Is there a point
                                  > you're trying to make?
                                  >
                                  > That's an interesting answer. I don't think of traceability
                                  > as a data structure as such. I think of it as something more
                                  > abstract--a concept that helps us understand the historical
                                  > links between one set of ideas and another, or an answer to a
                                  > question in the general form of "Why did we do that?"

                                  You're right. "Traceability" is a property. A "trace" is
                                  a data structure. It is something like a doubly-linked list
                                  between requirements and test cases, except that every test
                                  can support multiple requirements, and every requirement can
                                  be covered by multiple tests. I can't think of what such a
                                  data structure is called at the moment.


                                  Cheers,
                                  Todd.
                                • Bradley, Todd
                                  ... Yes! That s what I was trying to get at. The test cases should contain the traceability to the require- ments in some way. And the traceability matrix
                                  Message 16 of 16 , Mar 17, 2008
                                  View Source
                                  • 0 Attachment
                                    John O wrote:
                                    > Why can't the RTVM be THE test cases, or at least a
                                    > sub-set of the cases?

                                    Yes! That's what I was trying to get at. The test
                                    cases should contain the traceability to the require-
                                    ments in some way. And the traceability matrix is
                                    the human readable document generated from that data.


                                    Todd.
                                  Your message has been successfully submitted and would be delivered to recipients shortly.