Loading ...
Sorry, an error occurred while loading the content.
 

Re: how to show that junit works (validating junit)

Expand Messages
  • j_cumps
    Maybe you can show them the results of the JUnit unit tests (a.k.a. recursive certification). I ve uploaded a report of JUnit 4.1 tests to
    Message 1 of 10 , Sep 1, 2006
      Maybe you can show them the results of the JUnit unit tests (a.k.a.
      recursive certification).

      I've uploaded a report of JUnit 4.1 tests to
      http://junitpdfreport.sourceforge.net/examples/junit-junit.4.1.pdf

      Regards, Jan

      --- In junit@yahoogroups.com, "mvogelbusedge" <mvogel@...> wrote:
      >
      > In many parts of the pharmaceutial business applications must
      > be 'validated' based on the CFR 21 Part 11 standards. So our
      > customers sometimes don't want us to use an open source tool like
      > junit because it isn't formally validated, e.g., doesn't have written
      > requirements, traceable to design, traceable to unit and system
      tests,
      > with the output of the tests captured. Any suggestions for how to
      > address this other than telling them that junit is so extensively
      used
      > it must work, e.g., is there a suite of junit acceptance tests or
      even
      > unit tests that we can run to prove that it works according to its
      > specs?
      >
    • mvogelbusedge
      It is a weaker definition of formally validated where each system is considered separately. Minimally, the app must have requirements doc signed by
      Message 2 of 10 , Sep 4, 2006
        It is a weaker definition of formally validated where each system is
        considered separately. Minimally, the app must have requirements
        doc signed by user's/vendor, traceability to design documents from
        requirements, traceability from requirements to unit, system and
        user acceptance tests, plus a varitey of additional higher level
        validation strategy and test plan documents. Usually the
        traceability is shown as a traceability matrix. The evidence of all
        the tests and dev processes must be archived, signed, etc. This is
        mostly just standard good software development practice but ideally
        I need all these elements for junit to make the QA police
        comfortable. Their great fear is an FDA auditor shows up and asks
        for these artifacts for every system used by whatever they are
        auditing (such audits are semi-regular events). Vendor products,
        e.g., commercial test tools, have their software process and
        artifacts audited by the company QA department.

        No auditor could follow the CVS commits as a traceability matrix.
        What doc do you consider the standard users manual? The articles
        like the Cookbook don't cover all the features so they aren't easy
        to use as the basis of traceability. I'd probably need to use the
        Cooks Tour as the design document but I don't think it covers
        everything.

        --- In junit@yahoogroups.com, Robert Wenner <robert.wenner@...>
        wrote:
        > > So our
        > > customers sometimes don't want us to use an open source tool like
        > > junit because it isn't formally validated,
        >
        > Makes me curious: what tool is formally validated?
        > Can it even be formally validated without the underlying OS being
        formally
        > validated?
        >
        > > e.g., doesn't have written requirements,
        >
        > Give them the user's manual as requirements design document?
        >
        > > traceable to design, traceable to unit and system tests,
        >
        > Requirements to design tracing: You may be able to pull that from
        the CVS
        > or Subverson history, if the commits happened in small step.
        >
        > > with the output of the tests captured.
        >
        > Run the test with ant and the testreport task.
        > Commit the output to the repository to show them the history of
        succeeding
        > tests.
        >
        > > Any suggestions for how to
        > > address this other than telling them that junit is so
        extensively used
        > > it must work, e.g., is there a suite of junit acceptance tests
        or even
        > > unit tests that we can run to prove that it works according to
        its
        > > specs?
        >
        > I'd ask for the test documentation for one of the formally
        validated
        > tools. If they can actually provide such a thing, you may get a
        better
        > idea what they are looking for.
        >
        > Good luck, and please let us know what happened.
        >
        > Robert
        >
      • Antti Karanta
        ... I have no experience in developing software w/ this degree of formalism, and this post made me curious: I d understand that all the validation is required
        Message 3 of 10 , Sep 4, 2006
          > -----Original Message-----
          > From: junit@yahoogroups.com [mailto:junit@yahoogroups.com] On
          > Behalf Of mvogelbusedge
          > Sent: 4. syyskuuta 2006 16:54
          > To: junit@yahoogroups.com
          > Subject: [junit] Re: how to show that junit works (validating junit)
          >
          > It is a weaker definition of formally validated where each system is
          > considered separately. Minimally, the app must have requirements
          > doc signed by user's/vendor, traceability to design documents from
          > requirements, traceability from requirements to unit, system and
          > user acceptance tests, plus a varitey of additional higher level
          > validation strategy and test plan documents. Usually the
          > traceability is shown as a traceability matrix. The evidence of all
          > the tests and dev processes must be archived, signed, etc. This is
          > mostly just standard good software development practice but ideally
          > I need all these elements for junit to make the QA police
          > comfortable.

          I have no experience in developing software w/ this degree of
          formalism, and this post made me curious: I'd understand that all the
          validation is required of the components included w/ the software, but I
          don't really understand that it is required of the development tools,
          too. Is it required that the debugger is validated? The compiler? The
          IDE? The OS? The JVM? Actually, the OS and the JVM are pretty important
          for the correct functioning of the software, as is the compiler.

          As I see it, unit tests are a development tool - they aren't (usually)
          shipped w/ the software. You could, in principle, write and use unit
          tests, but keep their sources and binaries in a separate place and never
          even tell the auditors they exist, and the auditors would not know to
          miss them. I'm not suggesting you do this, I'm just a little confused as
          to why a development tool needs to be validated.
          Thenagain, if you want to use junit to validate your software
          formally, then I guess the validation tool itself needs to be validated?



          -Antti-
        • mvogelbusedge
          ... I ... important ... never ... as ... validated? ... They don t see any of our dev tools so they don t care about them being validated. They want the test
          Message 4 of 10 , Sep 5, 2006
            --- In junit@yahoogroups.com, "Antti Karanta" <Antti.Karanta@...> wrote:
            >
            > I have no experience in developing software w/ this degree of
            > formalism, and this post made me curious: I'd understand that all the
            > validation is required of the components included w/ the software, but
            I
            > don't really understand that it is required of the development tools,
            > too. Is it required that the debugger is validated? The compiler? The
            > IDE? The OS? The JVM? Actually, the OS and the JVM are pretty
            important
            > for the correct functioning of the software, as is the compiler.
            >
            > As I see it, unit tests are a development tool - they aren't (usually)
            > shipped w/ the software. You could, in principle, write and use unit
            > tests, but keep their sources and binaries in a separate place and
            never
            > even tell the auditors they exist, and the auditors would not know to
            > miss them. I'm not suggesting you do this, I'm just a little confused
            as
            > to why a development tool needs to be validated.
            > Thenagain, if you want to use junit to validate your software
            > formally, then I guess the validation tool itself needs to be
            validated?
            >
            >
            >
            > -Antti-
            >


            They don't see any of our dev tools so they don't care about them being
            validated. They want the test tools validated since these are what
            they use to validate the results of our use of the dev tools. They
            typically have strict rules about what versions of the JVM, OS, App
            Server, Database product, etc., can be used at run-time. They don't
            validate the JVM or OS but do some kind of validation of the dev process
            of the products that run on them, e.g., they would have audited BEA if
            using Weblogic.

            The auditors want to see the evidence of unit, integration, and user
            acceptance testing. So we could hide the tools used but when they saw
            automated test evidence they could question the validation status of the
            tools used. We also have to create various test plans (Master, Unit,
            User, etc.) that document how the testing was planned. The unit test
            plans I've written state that we use xUnit, how the tests are organized
            and how to find the test source code since it is the definition of the
            unit tests done.

            I'm not saying I agree with the need to validate junit since it is very
            likely better tested than most software artifacts in existence.
            However, if I could provide them with the things the QA folks and
            auditors are used to getting (requirements, traceability, test evidence)
            then their reluctance to let us use junit would go away. Last project I
            got away with using junit by documenting a series of manual tests that
            paralleled the junit coded tests and showing the results were the same.

            So what I need is:

            * Something to serve as requirements for junit * Something to
            serve as the design document for junit * unit tests of the things in
            the design doc (should be just the unit tests from a build) *
            integration/acceptance tests (could be my manual re-run of some of our
            project specific unit tests plus the tests that come with a
            distribution) * Some way to describe the traceability from the
            requirements to the design, requirements to acceptance tests. Doesn't
            need to be an actual matrix, could be just a description of the mapping.

            With enough work I could create this but I'm looking for help picking
            the best artifacts and approaches to minimize my work and so that it
            doesn't need to be redone with each release of junit.



            [Non-text portions of this message have been removed]
          • David Saff
            ... Sorry to come late to the party here. How has this effort gone? What strategy did you choose? At the time, it occurred to me that in the time it would
            Message 5 of 10 , Sep 21, 2006
              mvogelbusedge wrote:
              > I'm not saying I agree with the need to validate junit since it is very
              > likely better tested than most software artifacts in existence.
              > However, if I could provide them with the things the QA folks and
              > auditors are used to getting (requirements, traceability, test evidence)
              > then their reluctance to let us use junit would go away. Last project I
              > got away with using junit by documenting a series of manual tests that
              > paralleled the junit coded tests and showing the results were the same.
              >
              > So what I need is:
              >
              > * Something to serve as requirements for junit * Something to
              > serve as the design document for junit * unit tests of the things in
              > the design doc (should be just the unit tests from a build) *
              > integration/acceptance tests (could be my manual re-run of some of our
              > project specific unit tests plus the tests that come with a
              > distribution) * Some way to describe the traceability from the
              > requirements to the design, requirements to acceptance tests. Doesn't
              > need to be an actual matrix, could be just a description of the mapping.
              >
              > With enough work I could create this but I'm looking for help picking
              > the best artifacts and approaches to minimize my work and so that it
              > doesn't need to be redone with each release of junit.
              >

              Sorry to come late to the party here. How has this effort gone? What
              strategy did you choose? At the time, it occurred to me that in the
              time it would take to generate the documentation you specified, you
              could have a mutually-trusted developer inspect the entire applicable
              JUnit code base, but that might not be standard operating procedure.

              David Saff
            • mvogelbusedge
              ... At my last customer project we have worked our way around this problem by a lot of talking and doing some minimal manual testing to show junit works.
              Message 6 of 10 , Sep 23, 2006
                --- In junit@yahoogroups.com, David Saff <saff@...> wrote:
                At my last customer project we have worked our way around this problem
                by a lot of talking and doing some minimal manual testing to show
                junit works. Doing an inspection doesn't fit the FDA mandated model
                so I'm going to continue to work on assembing something resembling
                requirements, tests and test evidence with reasonable traceability.
                It won't be a priority till the next development project starts so I
                don't have details to share yet.

                > Sorry to come late to the party here. How has this effort gone?
                What
                > strategy did you choose? At the time, it occurred to me that in the
                > time it would take to generate the documentation you specified, you
                > could have a mutually-trusted developer inspect the entire
                applicable
                > JUnit code base, but that might not be standard operating procedure.
                >
                > David Saff
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.