Loading ...
Sorry, an error occurred while loading the content.

Re: Improving unit and system testing

Expand Messages
  • ralphvanroosmalen
    Hi, We had also some problem with unit tests, it was not part of the daily operation and the developers didn t fix the source daily. We solved this problem by:
    Message 1 of 20 , Oct 31, 2006
    • 0 Attachment
      Hi,

      We had also some problem with unit tests, it was not part of the daily
      operation and the developers didn't fix the source daily. We solved
      this problem by:

      1. Management support, our CTO is 100% convinced of the advantages of
      unit testing and he accepts the "delay" because of the unit testing.
      If you management does not support unit testing, it is getting difficult.

      2. Visibility, until a year ago we reported the unit results every day
      by email. This doesn't work, it is not visible for the teams and
      management and no one feels responsible. The solution was to install a
      pc in the corridor with a excel sheet that displays the unit results
      of last night and also the open problems in our problem registration
      system. Every one who goes to the toilet or coffee corner sees the
      results.

      3. A clear and supported work procedure by the teams. Our daily
      procedure is first fix unit problems, fix problems and as last build
      new functionality. It can be the task of the scrummaster to keep the
      team focused on the daily procedure.

      We improved our test process by using a product risk matrix. Our risk
      matrix has four quadrants and in every quadrant we specified the test
      activities. The highest risk has for example 90% unit coverage, review
      of unit test and test cases by other team member, high automation of
      test cases, review of source code etc. The lowest risk item have only
      50% unit coverage and no test automation.

      If you would like to have some more information please let me know.
      Bye
      Ralph

      QA Manager
      Planon - www.planon-fm.com

      --- In scrumdevelopment@yahoogroups.com, "Svetlana Didinchuk"
      <sdidinchuk@...> wrote:
      >
      > Hi,
      >
      >
      >
      > I have been recently assigned with the task to improve unit and system
      > testing process on my new team. We are not agile yet but want to go
      > there with one project team currently applying "some" agile methods and
      > everybody else watching the results. So I need to improve testing
      > process and therefore quality for both agile and not-agile teams.
      >
      >
      >
      > Currently unit testing is a developer responsibility and judging by the
      > number of items returned by QA or even worse by customers it does not
      > happen as it should. I know in part this is a result of poor planning
      > and pressure to deliver to a customer sooner but even though what can we
      > do to improve quality?
      >
      > Can you point me to some sources for better testing process and quality
      > improvement? Also I would appreciate any advice on this matter.
      >
      >
      >
      > Thanks,
      >
      >
      >
      > Svetlana Didinchuk
      >
    • Svetlana Didinchuk
      Thank you all for your help and recommendations on improving unit testing and quality! This is so great to be able to leverage experience of so many
      Message 2 of 20 , Nov 1, 2006
      • 0 Attachment

        Thank you all for your help and recommendations on improving unit testing and quality! This is so great to be able to leverage experience of so many professionals in this industry. Now I have what I need to tackle this problem (and our management too :-))

         

        I still have one question on automated unit testing though. Currently our product QA using Mercury for system tests automation and I wonder if developers can use the same tool for functionality unit testing.

         

        Have anybody used Mercury as a development unit testing automation tool?

        And another question: when you just starting to introduce automated unit testing for functionality as a part of development how long is the learning curve? The problem I am having now with the management (and even other lead developers) that yes, it sounds good but we cannot afford that long learning curve (for Mercury it was estimated for 129 days).

         

        I understand that Mercury is very powerful but also complicated. Are there tools which are easier and quicker to use? For example I have looked at Fit and FitNesse a little. Are they easy to use? How long is it for an average developer to learn how to write tests in it?

         

        Thanks in advance for your help,

         

         

        Svetlana Didinchuk

        sdidinchuk@...

        Software Engineer, Product Development

         

        Phoenix Interactive Design Inc.

        General Phone:  (519) 679-2913 ext. 4368

        http://www.phoenix-interactive.com

         


        From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of John Lee
        Sent: October 31, 2006 4:48 PM
        To: scrumdevelopment@yahoogroups.com
        Subject: RE: [scrumdevelopment] Improving unit and system testing

         

        Svetlana,

                    Yes, the focus should be unit testing and unit test automaton infrastructure as first order of business.  Once you have the unit testing plus QA process hashed out, look at how to implement the continuous integration model.   In this model, you have many components, but at very high level, these are what you need:

         

        1.      checkin validation to never break the mainline (build and sanity test based on distributed queue model) – automate to kick out checkins that does not pass this criteria – of cause, you need to have build-in exceptions to deal with component dependency issues.

        2.      nightly lights-out load plus feature specific testing to generate performance and system capacity trending data

        3.      nightly lights-out under load comparative analysis at functionality level performance of last released version verses the release in progress.

        4.      weekly (whatever duration that sense for your scrum) capacity edge testing plus longevity testing (duration of up to 48 hours or as it makes sense for you)

        5.      many more architectural analysis components, but if you do 1-4 well, you are mostly there.

         

         

        Thanks,

         

        John Lee / Director, System Test

        Desk: 415.536.8084

        Cell:   425.246.1223

        YIM: jlee_sfdc

        Salesforce.com

        One Market Street,  San Francisco , CA 94105

         

         

         

         


        From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
        Sent: Tuesday, October 31, 2006 1:08 PM
        To: scrumdevelopment@yahoogroups.com
        Subject: RE: [scrumdevelopment] Improving unit and system testing

         

        Hi John,

         

        >It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if >relevant.

         

        I don’t know if our software development is a typical one. We have product team writing and maintaining a base product in C++. Each release is going through QA system test. We also have professional services team. This team takes base product release and adds customizations to satisfy a particular customer implementation. Before it was a custom code in C++ but recently this team does only XML configuration changes.

         

        The nature of our product is that a lot of features are driven by XML: interface, functionality flow, language, print forms, etc. So even it is not strictly “development” this is still ‘code’ and it greatly affects functionality.

         

        I was transferred from product to this team 2 days ago and one of the reasons was to help them improve their process. Right now unit testing is up to developer without any guidelines. It is supposed it happens but nobody knows to what extent. Practice shows that it is not sufficient and issues are coming back from customers. These custom projects get to QA, they do regression test and system integrated test with customer. QA did not use automated tests and only consider using them now (even though base QA has been using automated testing for ages!). Because the process is waterfall QA gets the release in the end so all found issues need to be added hastily…you know the nightmare!

         

        I think QA needs to perform high-level testing ensuring that the system satisfies client’s requirements. For me it means that better unit testing is vital!

         

        Thanks,

         

        Svetlana Didinchuk

        Software Engineer

        Phoenix Interactive

        sdidinchuk@phoenix- interactive. com

        (519) 679 2913 ext. 4368

         

         

         


        From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of John Lee
        Sent: October 31, 2006 1:12 PM
        To: scrumdevelopment@ yahoogroups. com
        Subject: RE: [scrumdevelopment] Improving unit and system testing

         

        It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if relevant.

         

         

        Thanks,

         

        John Lee / Director, System Test

        Desk: 415.536.8084

        Cell:   425.246.1223

        YIM: jlee_sfdc

        Salesforce.com

        One Market Street ,  San Francisco , CA 94105

         

         

         

         


        From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of Svetlana Didinchuk
        Sent: Tuesday, October 31, 2006 6:50 AM
        To: scrumdevelopment@ yahoogroups. com
        Subject: [scrumdevelopment] Improving unit and system testing

         

        Hi,

         

        I have been recently assigned with the task to improve unit and system testing process on my new team. We are not agile yet but want to go there with one project team currently applying “some” agile methods and everybody else watching the results. So I need to improve testing process and therefore quality for both agile and not-agile teams.

         

        Currently unit testing is a developer responsibility and judging by the number of items returned by QA or even worse by customers it does not happen as it should. I know in part this is a result of poor planning and pressure to deliver to a customer sooner but even though what can we do to improve quality?

        Can you point me to some sources for better testing process and quality improvement? Also I would appreciate any advice on this matter.

         

        Thanks,

         

        Svetlana Didinchuk

      • Svetlana Didinchuk
        Hi Ralph, Thanks for your response. Can you give me more details on this product risk matrix? Sounds as a good approach for testing... Thanks, Svetlana
        Message 3 of 20 , Nov 1, 2006
        • 0 Attachment
          Hi Ralph,

          Thanks for your response.

          Can you give me more details on this product risk matrix? Sounds as a good approach for testing...

          Thanks,

          Svetlana Didinchuk
          sdidinchuk@...
          Software Engineer, Product Development

          Phoenix Interactive Design Inc.
          General Phone:  (519) 679-2913 ext. 4368
          http://www.phoenix-interactive.com


          -----Original Message-----
          From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of ralphvanroosmalen
          Sent: November 1, 2006 2:59 AM
          To: scrumdevelopment@yahoogroups.com
          Subject: [scrumdevelopment] Re: Improving unit and system testing

          Hi,

          We had also some problem with unit tests, it was not part of the daily
          operation and the developers didn't fix the source daily. We solved
          this problem by:

          1. Management support, our CTO is 100% convinced of the advantages of
          unit testing and he accepts the "delay" because of the unit testing.
          If you management does not support unit testing, it is getting difficult.

          2. Visibility, until a year ago we reported the unit results every day
          by email. This doesn't work, it is not visible for the teams and
          management and no one feels responsible. The solution was to install a
          pc in the corridor with a excel sheet that displays the unit results
          of last night and also the open problems in our problem registration
          system. Every one who goes to the toilet or coffee corner sees the
          results.

          3. A clear and supported work procedure by the teams. Our daily
          procedure is first fix unit problems, fix problems and as last build
          new functionality. It can be the task of the scrummaster to keep the
          team focused on the daily procedure.

          We improved our test process by using a product risk matrix. Our risk
          matrix has four quadrants and in every quadrant we specified the test
          activities. The highest risk has for example 90% unit coverage, review
          of unit test and test cases by other team member, high automation of
          test cases, review of source code etc. The lowest risk item have only
          50% unit coverage and no test automation.

          If you would like to have some more information please let me know.
          Bye
          Ralph

          QA Manager
          Planon - www.planon-fm.com

          --- In scrumdevelopment@yahoogroups.com, "Svetlana Didinchuk"
          <sdidinchuk@...> wrote:
          >
          > Hi,
          >
          >
          >
          > I have been recently assigned with the task to improve unit and system
          > testing process on my new team. We are not agile yet but want to go
          > there with one project team currently applying "some" agile methods and
          > everybody else watching the results. So I need to improve testing
          > process and therefore quality for both agile and not-agile teams.
          >
          >
          >
          > Currently unit testing is a developer responsibility and judging by the
          > number of items returned by QA or even worse by customers it does not
          > happen as it should. I know in part this is a result of poor planning
          > and pressure to deliver to a customer sooner but even though what can we
          > do to improve quality?
          >
          > Can you point me to some sources for better testing process and quality
          > improvement? Also I would appreciate any advice on this matter.
          >
          >
          >
          > Thanks,
          >
          >
          >
          > Svetlana Didinchuk
          >





          To Post a message, send it to: scrumdevelopment@...
          To Unsubscribe, send a blank message to: scrumdevelopment-unsubscribe@...
          Yahoo! Groups Links
        • Keith Ray
          Google will find a lot of stuff a bout pair programming, and Martin Fowler just mentioned it in his bliki. What doesn t matter is how well some computer
          Message 4 of 20 , Nov 1, 2006
          • 0 Attachment
            Google will find a lot of stuff a bout pair programming, and Martin
            Fowler just mentioned it in his bliki.

            What doesn't matter is how well some computer science students did in
            a study, or some just-"trained" professionals did in another study.
            What matters is how the programmers experience it. My experience is
            that productivity is about the same as two people working solo, but
            the code quality is better -- which will be important later on as you
            build on top of previously-written code.

            You can introduce it without giving it a name by asking "could you sit
            with me and help me work on this problem?" or "Can I sit with you and
            maybe offer some tips?"

            I hear that even the most pro-Pair-Programming shops only do pair
            programming 6 hours a day, or more typically 60%.

            The better classes/training/workshops on TDD will have the students
            working in pairs.

            I've seen academic papers that show how working in pairs in
            educational situations often get better results; sorry I can't find
            those papers now.

            > I was looking into pair programming and test-driven development as a way
            > to improve the quality. The problem with pair programming is that it is
            > such a scary thought to have two developers to work on the same task.

            It takes two people to move a large piece of furniture. It took both
            Gilbert and Sullivan to create all those operettas. Why do we assume
            it takes only one person to turn an idea into working tests and code?

            --

            C. Keith Ray
            <http://homepage.mac.com/keithray/blog/index.html>
            <http://homepage.mac.com/keithray/xpminifaq.html>
            <http://homepage.mac.com/keithray/resume2.html>
          • John Lee
            Mercury is not suitable for unit test automation. Try NUnit. Thanks, John Lee / Director, System Test Desk: 415.536.8084 Cell: 425.246.1223 YIM: jlee_sfdc
            Message 5 of 20 , Nov 1, 2006
            • 0 Attachment

              Mercury is not suitable for unit test automation.  Try NUnit. 

               

              Thanks,

               

              John Lee / Director, System Test

              Desk: 415.536.8084

              Cell:   425.246.1223

              YIM: jlee_sfdc

              Salesforce.com

              One Market Street,  San Francisco , CA 94105

               

               

               

               


              From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
              Sent: Wednesday, November 01, 2006 7:50 AM
              To: scrumdevelopment@yahoogroups.com
              Subject: RE: [scrumdevelopment] Improving unit and system testing

               

              Thank you all for your help and recommendations on improving unit testing and quality! This is so great to be able to leverage experience of so many professionals in this industry. Now I have what I need to tackle this problem (and our management too :-))

               

              I still have one question on automated unit testing though. Currently our product QA using Mercury for system tests automation and I wonder if developers can use the same tool for functionality unit testing.

               

              Have anybody used Mercury as a development unit testing automation tool?

              And another question: when you just starting to introduce automated unit testing for functionality as a part of development how long is the learning curve? The problem I am having now with the management (and even other lead developers) that yes, it sounds good but we cannot afford that long learning curve (for Mercury it was estimated for 129 days).

               

              I understand that Mercury is very powerful but also complicated. Are there tools which are easier and quicker to use? For example I have looked at Fit and FitNesse a little. Are they easy to use? How long is it for an average developer to learn how to write tests in it?

               

              Thanks in advance for your help,

               

               

              Svetlana Didinchuk

              sdidinchuk@phoenix- interactive. com

              Software Engineer, Product Development

               

              Phoenix Interactive Design Inc.

              General Phone:  (519) 679-2913 ext. 4368

              http://www.phoenix- interactive. com

               


              From: scrumdevelopment@ yahoogroups. com [mailto:scrumdevelo pment@yahoogroup s.com] On Behalf Of John Lee
              Sent: October 31, 2006 4:48 PM
              To: scrumdevelopment@ yahoogroups. com
              Subject: RE: [scrumdevelopment] Improving unit and system testing

               

              Svetlana,

                          Yes, the focus should be unit testing and unit test automaton infrastructure as first order of business.  Once you have the unit testing plus QA process hashed out, look at how to implement the continuous integration model.   In this model, you have many components, but at very high level, these are what you need:

               

              1.      checkin validation to never break the mainline (build and sanity test based on distributed queue model) – automate to kick out checkins that does not pass this criteria – of cause, you need to have build-in exceptions to deal with component dependency issues.

              2.      nightly lights-out load plus feature specific testing to generate performance and system capacity trending data

              3.      nightly lights-out under load comparative analysis at functionality level performance of last released version verses the release in progress.

              4.      weekly (whatever duration that sense for your scrum) capacity edge testing plus longevity testing (duration of up to 48 hours or as it makes sense for you)

              5.      many more architectural analysis components, but if you do 1-4 well, you are mostly there.

               

               

              Thanks,

               

              John Lee / Director, System Test

              Desk: 415.536.8084

              Cell:   425.246.1223

              YIM: jlee_sfdc

              Salesforce.com

              One Market Street ,  San Francisco , CA 94105

               

               

               

               


              From: scrumdevelopment@ yahoogroups. com [mailto:scrumdevelo pment@yahoogroup s.com] On Behalf Of Svetlana Didinchuk
              Sent: Tuesday, October 31, 2006 1:08 PM
              To: scrumdevelopment@ yahoogroups. com
              Subject: RE: [scrumdevelopment] Improving unit and system testing

               

              Hi John,

               

              >It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if >relevant.

               

              I don’t know if our software development is a typical one. We have product team writing and maintaining a base product in C++. Each release is going through QA system test. We also have professional services team. This team takes base product release and adds customizations to satisfy a particular customer implementation. Before it was a custom code in C++ but recently this team does only XML configuration changes.

               

              The nature of our product is that a lot of features are driven by XML: interface, functionality flow, language, print forms, etc. So even it is not strictly “development” this is still ‘code’ and it greatly affects functionality.

               

              I was transferred from product to this team 2 days ago and one of the reasons was to help them improve their process. Right now unit testing is up to developer without any guidelines. It is supposed it happens but nobody knows to what extent. Practice shows that it is not sufficient and issues are coming back from customers. These custom projects get to QA, they do regression test and system integrated test with customer. QA did not use automated tests and only consider using them now (even though base QA has been using automated testing for ages!). Because the process is waterfall QA gets the release in the end so all found issues need to be added hastily…you know the nightmare!

               

              I think QA needs to perform high-level testing ensuring that the system satisfies client’s requirements. For me it means that better unit testing is vital!

               

              Thanks,

               

              Svetlana Didinchuk

              Software Engineer

              Phoenix Interactive

              sdidinchuk@phoenix- interactive. com

              (519) 679 2913 ext. 4368

               

               

               


              From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of John Lee
              Sent: October 31, 2006 1:12 PM
              To: scrumdevelopment@ yahoogroups. com
              Subject: RE: [scrumdevelopment] Improving unit and system testing

               

              It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if relevant.

               

               

              Thanks,

               

              John Lee / Director, System Test

              Desk: 415.536.8084

              Cell:   425.246.1223

              YIM: jlee_sfdc

              Salesforce.com

              One Market Street ,  San Francisco , CA 94105

               

               

               

               


              From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of Svetlana Didinchuk
              Sent: Tuesday, October 31, 2006 6:50 AM
              To: scrumdevelopment@ yahoogroups. com
              Subject: [scrumdevelopment] Improving unit and system testing

               

              Hi,

               

              I have been recently assigned with the task to improve unit and system testing process on my new team. We are not agile yet but want to go there with one project team currently applying “some” agile methods and everybody else watching the results. So I need to improve testing process and therefore quality for both agile and not-agile teams.

               

              Currently unit testing is a developer responsibility and judging by the number of items returned by QA or even worse by customers it does not happen as it should. I know in part this is a result of poor planning and pressure to deliver to a customer sooner but even though what can we do to improve quality?

              Can you point me to some sources for better testing process and quality improvement? Also I would appreciate any advice on this matter.

               

              Thanks,

               

              Svetlana Didinchuk

            • Chris S. Sterling
              The automation of functional tests are sometimes referred to as acceptance tests or more recently customer tests . There is some more detailed information
              Message 6 of 20 , Nov 1, 2006
              • 0 Attachment
                The automation of functional tests are sometimes referred to as "acceptance tests" or more recently "customer tests".  There is some more detailed information on this at:
                 
                I have been on teams that have used Rational Robot and Mercury Quick Test Pro to automate functional tests.  The issues that I have seen is the learning curve for these tools as you have pointed out.  Also, they were not designed with the recognition of TDD methods such as writing a failing test first.
                 
                We developed a tool based on Fit, FitNesse, and Selenium which we call StoryTestIQ (http://storytestiq.sourceforge.net/) which we use and update on a regular basis.  We have approximately 7 projects using StoryTestIQ (or "STIQ" for short) at SolutionsIQ and many of these projects develop failing acceptance tests before implementing the actual code to satisfy the tests.  The definition of these "executable requirements" for our Sprint are created after the Sprint Planning Meeting based on acceptance criteria identified by the Product Owner.  We have found that including subject matter experts such as the Product Owner, business analysts, and end users into the development of the automated acceptance tests has increased our productivity and customer satisfaction.  Our productivity gains are due in part to well defined acceptance criteria that allows our teams to focus on what the customer values most for a requirement/feature/user story.
                 
                We use the syntax set out by Selenium (http://www.openqa.org/) for STIQ and we have found that some Product Owners and business analysts are actually able to create these automated acceptance tests for those stories which have been identified as highest value on the Product Backlog.  The syntax is something like:
                 
                | assertTextPresent | Welcome to my site! | |
                | clickAndWait | button=Next Page | |
                 
                As you can see it is based on an HTML table with 3 columns and we use the wiki capabilities of FitNesse to edit the tests inside the web browser where they can readily be executed.  Since we use FitNesse underneath the covers, we also have the ability to run Fit fixtures which helps in situations where you are testing non web application functionality such as a web service.
                 
                Anyways, StoryTestIQ, FitNesse, Selenium, and Fit I have found are all valid options for automating acceptance tests.  StoryTestIQ and Selenium tend to be easier to work with initially in my experience for web applications.  Fit and FitNesse along with StoryTestIQ give more capabilities passed functional testing of web applications through Fit fixtures.  I hope this helps.
                 
                Chris Sterling
                 


                From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
                Sent: Wednesday, November 01, 2006 7:50 AM
                To: scrumdevelopment@yahoogroups.com
                Subject: RE: [scrumdevelopment] Improving unit and system testing

                Thank you all for your help and recommendations on improving unit testing and quality! This is so great to be able to leverage experience of so many professionals in this industry. Now I have what I need to tackle this problem (and our management too :-))

                I still have one question on automated unit testing though. Currently our product QA using Mercury for system tests automation and I wonder if developers can use the same tool for functionality unit testing.

                Have anybody used Mercury as a development unit testing automation tool?

                And another question: when you just starting to introduce automated unit testing for functionality as a part of development how long is the learning curve? The problem I am having now with the management (and even other lead developers) that yes, it sounds good but we cannot afford that long learning curve (for Mercury it was estimated for 129 days).

                I understand that Mercury is very powerful but also complicated. Are there tools which are easier and quicker to use? For example I have looked at Fit and FitNesse a little. Are they easy to use? How long is it for an average developer to learn how to write tests in it?

                Thanks in advance for your help,

                Svetlana Didinchuk

                sdidinchuk@phoenix- interactive. com

                Software Engineer, Product Development

                Phoenix Interactive Design Inc.

                General Phone:  (519) 679-2913 ext. 4368

                http://www.phoenix- interactive. com


                From: scrumdevelopment@ yahoogroups. com [mailto:scrumdevelo pment@yahoogroup s.com] On Behalf Of John Lee
                Sent: October 31, 2006 4:48 PM
                To: scrumdevelopment@ yahoogroups. com
                Subject: RE: [scrumdevelopment] Improving unit and system testing

                Svetlana,

                            Yes, the focus should be unit testing and unit test automaton infrastructure as first order of business.  Once you have the unit testing plus QA process hashed out, look at how to implement the continuous integration model.   In this model, you have many components, but at very high level, these are what you need:

                1.      checkin validation to never break the mainline (build and sanity test based on distributed queue model) – automate to kick out checkins that does not pass this criteria – of cause, you need to have build-in exceptions to deal with component dependency issues.

                2.      nightly lights-out load plus feature specific testing to generate performance and system capacity trending data

                3.      nightly lights-out under load comparative analysis at functionality level performance of last released version verses the release in progress.

                4.      weekly (whatever duration that sense for your scrum) capacity edge testing plus longevity testing (duration of up to 48 hours or as it makes sense for you)

                5.      many more architectural analysis components, but if you do 1-4 well, you are mostly there.

                Thanks,

                John Lee / Director, System Test

                Desk: 415.536.8084

                Cell:   425.246.1223

                YIM: jlee_sfdc

                Salesforce.com

                One Market Street,  San Francisco , CA 94105


                From: scrumdevelopment@ yahoogroups. com [mailto:scrumdevelo pment@yahoogroup s.com] On Behalf Of Svetlana Didinchuk
                Sent: Tuesday, October 31, 2006 1:08 PM
                To: scrumdevelopment@ yahoogroups. com
                Subject: RE: [scrumdevelopment] Improving unit and system testing

                Hi John,

                >It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if

                >relevant.

                I don’t know if our software development is a typical one. We have product team writing and maintaining a base product in C++. Each release is going through QA system test. We also have professional services team. This team takes base product release and adds customizations to satisfy a particular customer implementation. Before it was a custom code in C++ but recently this team does only XML configuration changes.

                The nature of our product is that a lot of features are driven by XML: interface, functionality flow, language, print forms, etc. So even it is not strictly “development” this is still ‘code’ and it greatly affects functionality.

                I was transferred from product to this team 2 days ago and one of the reasons was to help them improve their process. Right now unit testing is up to developer without any guidelines. It is supposed it happens but nobody knows to what extent. Practice shows that it is not sufficient and issues are coming back from customers. These custom projects get to QA, they do regression test and system integrated test with customer. QA did not use automated tests and only consider using them now (even though base QA has been using automated testing for ages!). Because the process is waterfall QA gets the release in the end so all found issues need to be added hastily…you know the nightmare!

                I think QA needs to perform high-level testing ensuring that the system satisfies client’s requirements. For me it means that better unit testing is vital!

                Thanks,

                Svetlana Didinchuk

                Software Engineer

                Phoenix Interactive

                sdidinchuk@phoenix- interactive. com

                (519) 679 2913 ext. 4368


                From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of John Lee
                Sent: October 31, 2006 1:12 PM
                To: scrumdevelopment@ yahoogroups. com
                Subject: RE: [scrumdevelopment] Improving unit and system testing

                It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if relevant.

                Thanks,

                John Lee / Director, System Test

                Desk: 415.536.8084

                Cell:   425.246.1223

                YIM: jlee_sfdc

                Salesforce.com

                One Market Street ,  San Francisco , CA 94105


                From: scrumdevelopment@ yahoogroups. com [mailto: scrumdevelopment@ yahoogroups. com ] On Behalf Of Svetlana Didinchuk
                Sent: Tuesday, October 31, 2006 6:50 AM
                To: scrumdevelopment@ yahoogroups. com
                Subject: [scrumdevelopment] Improving unit and system testing

                Hi,

                I have been recently assigned with the task to improve unit and system testing process on my new team. We are not agile yet but want to go there with one project team currently applying “some” agile methods and everybody else watching the results. So I need to improve testing process and therefore quality for both agile and not-agile teams.

                Currently unit testing is a developer responsibility and judging by the number of items returned by QA or even worse by customers it does not happen as it should. I know in part this is a result of poor planning and pressure to deliver to a customer sooner but even though what can we do to improve quality?

                Can you point me to some sources for better testing process and quality improvement? Also I would appreciate any advice on this matter.

                Thanks,

                Svetlana Didinchuk

              • Alex Pukinskis
                Mercury has several different products for testing; all were designed with the idea that tests would be written in big bursts and then executed in big bursts.
                Message 7 of 20 , Nov 2, 2006
                • 0 Attachment
                  Re: [scrumdevelopment] Improving unit and system testing Mercury has several different products for testing; all were designed with the idea that tests would be written in big bursts and then executed in big bursts.  As such, they’re not great for projects where requirements evolve continuously.  This is true of most traditional commercial products, and the reason so many in the agile movement recommend open source tools.  

                  In the last few years we’ve started to see more commercial testing tools that are aligned with the agile approach, but it seems like the open source ones are still better, for the moment.

                  -Alex

                  On 11/1/06 7:08 PM, "John Lee" <jlee@...> wrote:

                  Mercury is not suitable for unit test automation.  Try NUnit.  
                   

                  Thanks,

                  John Lee / Director, System Test
                  Desk: 415.536.8084
                  Cell:   425.246.1223
                  YIM: jlee_sfdc
                  Salesforce.com
                  One Market Street,  San Francisco, CA 94105

                   
                   
                   


                  From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
                  Sent: Wednesday, November 01, 2006 7:50 AM
                  To: scrumdevelopment@yahoogroups.com
                  Subject: RE: [scrumdevelopment] Improving unit and system testing


                  Thank you all for your help and recommendations on improving unit testing and quality! This is so great to be able to leverage experience of so many professionals in this industry. Now I have what I need to tackle this problem (and our management too :-))
                   
                  I still have one question on automated unit testing though. Currently our product QA using Mercury for system tests automation and I wonder if developers can use the same tool for functionality unit testing.
                   
                  Have anybody used Mercury as a development unit testing automation tool?
                  And another question: when you just starting to introduce automated unit testing for functionality as a part of development how long is the learning curve? The problem I am having now with the management (and even other lead developers) that yes, it sounds good but we cannot afford that long learning curve (for Mercury it was estimated for 129 days).
                   
                  I understand that Mercury is very powerful but also complicated. Are there tools which are easier and quicker to use? For example I have looked at Fit and FitNesse a little. Are they easy to use? How long is it for an average developer to learn how to write tests in it?
                   
                  Thanks in advance for your help,

                   

                  Svetlana Didinchuk
                  sdidinchuk@...
                  Software Engineer, Product Development

                  Phoenix Interactive Design Inc.
                  General Phone:  (519) 679-2913 ext. 4368
                  http://www.phoenix-interactive.com <http://www.phoenix-interactive.com/>


                  From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of John Lee
                  Sent: October 31, 2006 4:48 PM
                  To: scrumdevelopment@yahoogroups.com
                  Subject: RE: [scrumdevelopment] Improving unit and system testing

                  Svetlana,
                             Yes, the focus should be unit testing and unit test automaton infrastructure as first order of business.  Once you have the unit testing plus QA process hashed out, look at how to implement the continuous integration model.  In this model, you have many components, but at very high level, these are what you need:
                   
                  1.
                       checkin validation to never break the mainline (build and sanity test based on distributed queue model) – automate to kick out checkins that does not pass this criteria – of cause, you need to have build-in exceptions to deal with component dependency issues.
                  2.
                       nightly lights-out load plus feature specific testing to generate performance and system capacity trending data
                  3.
                       nightly lights-out under load comparative analysis at functionality level performance of last released version verses the release in progress.
                  4.
                       weekly (whatever duration that sense for your scrum) capacity edge testing plus longevity testing (duration of up to 48 hours or as it makes sense for you)
                  5.
                       many more architectural analysis components, but if you do 1-4 well, you are mostly there.
                   
                   

                  Thanks,

                  John Lee / Director, System Test
                  Desk: 415.536.8084
                  Cell:   425.246.1223
                  YIM: jlee_sfdc
                  Salesforce.com
                  One Market Street,  San Francisco, CA 94105

                   
                   
                   


                  From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
                  Sent: Tuesday, October 31, 2006 1:08 PM
                  To: scrumdevelopment@yahoogroups.com
                  Subject: RE: [scrumdevelopment] Improving unit and system testing


                  Hi John,
                   
                  >It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if >relevant.
                   
                  I don’t know if our software development is a typical one. We have product team writing and maintaining a base product in C++. Each release is going through QA system test. We also have professional services team. This team takes base product release and adds customizations to satisfy a particular customer implementation. Before it was a custom code in C++ but recently this team does only XML configuration changes.
                   
                  The nature of our product is that a lot of features are driven by XML: interface, functionality flow, language, print forms, etc. So even it is not strictly “development” this is still ‘code’ and it greatly affects functionality.
                   
                  I was transferred from product to this team 2 days ago and one of the reasons was to help them improve their process. Right now unit testing is up to developer without any guidelines. It is supposed it happens but nobody knows to what extent. Practice shows that it is not sufficient and issues are coming back from customers. These custom projects get to QA, they do regression test and system integrated test with customer. QA did not use automated tests and only consider using them now (even though base QA has been using automated testing for ages!). Because the process is waterfall QA gets the release in the end so all found issues need to be added hastily…you know the nightmare!
                   
                  I think QA needs to perform high-level testing ensuring that the system satisfies client’s requirements. For me it means that better unit testing is vital!
                   
                  Thanks,
                   
                  Svetlana Didinchuk
                  Software Engineer
                  Phoenix Interactive
                  sdidinchuk@...
                  (519) 679 2913 ext. 4368





                  From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of John Lee
                  Sent: October 31, 2006 1:12 PM
                  To: scrumdevelopment@yahoogroups.com
                  Subject: RE: [scrumdevelopment] Improving unit and system testing

                  It seems there are as many definition of system testing, as there are opinions.  What is your current system testing process, and perhaps I can add my experience, if relevant.
                   
                   

                  Thanks,

                  John Lee / Director, System Test
                  Desk: 415.536.8084
                  Cell:   425.246.1223
                  YIM: jlee_sfdc
                  Salesforce.com
                  One Market Street,  San Francisco, CA 94105

                   
                   
                   


                  From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Svetlana Didinchuk
                  Sent: Tuesday, October 31, 2006 6:50 AM
                  To: scrumdevelopment@yahoogroups.com
                  Subject: [scrumdevelopment] Improving unit and system testing


                  Hi,
                   
                  I have been recently assigned with the task to improve unit and system testing process on my new team. We are not agile yet but want to go there with one project team currently applying “some” agile methods and everybody else watching the results. So I need to improve testing process and therefore quality for both agile and not-agile teams.
                   
                  Currently unit testing is a developer responsibility and judging by the number of items returned by QA or even worse by customers it does not happen as it should. I know in part this is a result of poor planning and pressure to deliver to a customer sooner but even though what can we do to improve quality?
                  Can you point me to some sources for better testing process and quality improvement? Also I would appreciate any advice on this matter.
                   
                  Thanks,
                   
                  Svetlana Didinchuk



                  --
                  Alex Pukinskis - Agile Coach
                  Rally Software Development - http://rallydev.com/

                • Jerrad Anderson
                  An alternative to fitnesse would be. http://atomicobject.com/pages/System+Testing+in+Ruby or
                  Message 8 of 20 , Nov 3, 2006
                  • 0 Attachment
                    An alternative to fitnesse would be.
                     
                     
                    or
                     
                     
                    What's nice about the above two is that it may be possible to leverage unit tests at the same time.
                     
                    cheers,
                     
                    Jerrad

                     
                    On 10/31/06, Svetlana Didinchuk <sdidinchuk@...> wrote:

                    Hi,

                     

                    I have been recently assigned with the task to improve unit and system testing process on my new team. We are not agile yet but want to go there with one project team currently applying "some" agile methods and everybody else watching the results. So I need to improve testing process and therefore quality for both agile and not-agile teams.

                     

                    Currently unit testing is a developer responsibility and judging by the number of items returned by QA or even worse by customers it does not happen as it should. I know in part this is a result of poor planning and pressure to deliver to a customer sooner but even though what can we do to improve quality?

                    Can you point me to some sources for better testing process and quality improvement? Also I would appreciate any advice on this matter.

                     

                    Thanks,

                     

                    Svetlana Didinchuk


                  Your message has been successfully submitted and would be delivered to recipients shortly.