Loading ...
Sorry, an error occurred while loading the content.

Test Coverage Not Execution Coverage

Expand Messages
  • Jay Flowers
    I have been tinkering with the idea of test coverage as apposed to code or execution coverage. Test coverage links the test fixtures to production classes and
    Message 1 of 2 , Feb 1, 2006
    • 0 Attachment
      I have been tinkering with the idea of test coverage as apposed to code or
      execution coverage. Test coverage links the test fixtures to production
      classes and test methods to production members. What does this show us you
      say? Lets take a look a an example report:


      Summary Assembly Count :1 Total Percent Coverage

      17/44 39%
      ------------------------------
      ------------------------------
      Assemblies
      ------------------------------
      Assembly Percent Coverage Members Covered Total Memebers
      plugin.dll<file:///C:/Projects/TestCoverage/DetailReport.html#plugin.dll>
      39%
      17 44 Assembly :plugin.dll Members Covered Total Members

      17/44 39%
      ------------------------------
      Covered Test Memeber

      GetEnumerator

      CopyTo
      tGetRegistrant GetRegistrant
      tGetRegistrant GetRegistrant
      tGetInstanceOf GetInstanceOf
      tGetInstanceOf GetInstanceOf
      tGetPluginType GetPluginType
      tAdd Add
      tRemove Remove
      tContains Contains
      tContains Contains

      AddRange

      RemoveRange

      Clear

      GetEnumerator

      .ctor

      Count

      IsSynchronized

      SyncRoot

      Item

      Reset

      MoveNext

      .ctor

      Current

      GetRegistrants

      .ctor
      tRefresh RefreshRegistry

      GetRegistry

      SaveRegistry

      ContainsRegistration
      tGetPluginTitle GetPluginTitle
      tGetPluginDescription GetPluginDescription

      Register
      tUnregister Unregister
      tGetPluginObject GetPluginObject

      .ctor
      tAbsolutePath GetInstance
      tGetPluginType GetPluginType

      LoadAssembly

      .ctor

      .ctor
      tRelativePath FullAssemblyPath

      Title

      Description


































































































      Now a Code Coverage report (be aware the bars don't appear to be rendering
      well in the email):

      Assembly Path :plugin.dll Total Coverage Total IL Instructions

      1014/1345 75%
      ------------------------------
      Coverage IL Count Function

      10/10 100% EF.Plugin.Registrar..cctor

      6/6 100% EF.Plugin.Registrar.GetRegistry

      11/11 100% EF.Plugin.Registrar.get_Registry

      36/36 100% EF.Plugin.Registrar.Reconstitute

      11/11 100% EF.Plugin.Registry..ctor

      10/10 100% EF.Plugin.Registry.GetEnumerator

      10/10 100% EF.Plugin.Registry.GetRegistrant

      49/91 53% EF.Plugin.Registry.GetRegistrant

      12/12 100% EF.Plugin.Registry.GetInstanceOf

      88/92 95% EF.Plugin.Registry.GetInstanceOf

      37/76 48% EF.Plugin.Registry.GetPluginType

      0/10 0% EF.Plugin.Registry.CopyTo

      9/9 100% EF.Plugin.Registry.get_Count

      0/6 0% EF.Plugin.Registry.get_IsSynchronized

      0/6 0% EF.Plugin.Registry.get_SyncRoot

      10/10 100% EF.Plugin.Registry.get_Item

      21/21 100% EF.Plugin.Registry.Add

      22/28 78% EF.Plugin.Registry.Remove

      39/41 95% EF.Plugin.Registry.Contains

      9/9 100% EF.Plugin.Registry.Contains

      46/57 80% EF.Plugin.Registry.AddRange

      50/61 81% EF.Plugin.Registry.RemoveRange

      13/13 100% EF.Plugin.Registry.Clear

      0/9 0% EF.Plugin.Registry.GetEnumerator

      5/5 100% EF.Plugin.Registration..ctor

      0/5 0% EF.Plugin.Registrar..ctor

      8/8 100% EF.Plugin.Registrar.RefreshRegistry

      26/36 72% EF.Plugin.Registrar.SaveRegistry

      47/47 100% EF.Plugin.Registrar.ContainsRegistration

      12/19 63% EF.Plugin.Registrar.LoadAssembly

      28/34 82% EF.Plugin.Registrar.GetPluginTitle

      28/32 87% EF.Plugin.Registrar.GetPluginDescription

      55/55 100% EF.Plugin.Registrar.Register

      55/55 100% EF.Plugin.Registrar.Unregister

      22/29 75% EF.Plugin.Registrar.GetPluginObject

      6/6 100% EF.Plugin.Registrant..ctor

      35/46 76% EF.Plugin.Registrant.get_FullAssemblyPath

      0/30 0% EF.Plugin.Registrant.get_Title

      0/28 0% EF.Plugin.Registrant.get_Description

      0/21 0% EF.Plugin.Registrant..ctor

      62/90 68% EF.Plugin.Registrant.GetInstance

      50/61 81% EF.Plugin.Registrant.GetPluginType

      9/9 100% EF.Plugin.Registrant.LoadAssembly

      12/18 66% EF.Plugin.Registrant.LoadAssembly












































































      So by the code coverage reporting 75% coverage I should feel confident about
      the health of the product, right? Well lets bring the test coverage report
      of only 39% into the picture. Now I am not so sure. It seems there is a
      lot of indirect testing. Is that okay? It might be. All by itself there
      is no problem but it is useful to know if you are trying to figure out why
      this part of the product is frequinty broken and there is high code
      coverage, it is all indirect testing. It can also be usful when trying to
      backfill unit tests. Code coverage is great for helping you get all those
      branches inside a member but it is a poor tool for determining test coverage
      of classes and members.

      What do you think?
      This is part where you shoot me down and tell me all those things that I did
      not think of or consider...

      --
      Jay Flowers
      ----------------------------------------------------------------------
      http://jayflowers.com
      ---------------------------------------------------------------------


      [Non-text portions of this message have been removed]
    • Charlie Poole
      Hi Jay, I ve done this for a client and it proved very useful. We used two methods to tie NUnit tests to the class being tested. For test fixtures that had a
      Message 2 of 2 , Feb 1, 2006
      • 0 Attachment
        Hi Jay,

        I've done this for a client and it proved very useful.

        We used two methods to tie NUnit tests to the class being tested. For test
        fixtures that had a one-to-one correspondence with production classes, we
        based it on a naming convention - SomeProductionClassTests, for example. For
        cases where we could not do that, we used the Description property of the
        TestFixture attribute. Just to make it readable, we scanned the description
        for the names of classes but a simpler implementation could just use the
        class name alone.

        We produced summaries by assembly class-level coverage within each assembly.
        We counted lines of code, number of tests and number of asserts. Rather than
        have some set of standards that had to be rigidly adhered to, we simply
        posted the results in the team area. Folks with only a few tests increased
        them, sometimes with a little prompting.

        Producing a two-level report required built-in knowledge of how tests and
        production code were distributed across assemblies - we had 60+ assemblies
        of which 8 to 10 were test assemblies. I can imagine doing this in a more
        general way, using some sort of table-driven approach, or one could simply
        report on one assembly at a time.

        Charlie

        > I have been tinkering with the idea of test coverage as
        > apposed to code or execution coverage. Test coverage links
        > the test fixtures to production classes and test methods to
        > production members. What does this show us you say? Lets
        > take a look a an example report:
        >
        >
        > Summary Assembly Count :1 Total Percent Coverage
        >
        > 17/44 39%
        > ------------------------------
        > ------------------------------
        > Assemblies
        > ------------------------------
        > Assembly Percent Coverage Members Covered Total Memebers
        > plugin.dll<file:///C:/Projects/TestCoverage/DetailReport.html#
        > plugin.dll>
        > 39%
        > 17 44 Assembly :plugin.dll Members Covered Total Members
        >
        > 17/44 39%
        > ------------------------------
        > Covered Test Memeber
        >
        > GetEnumerator
        >
        > CopyTo
        > tGetRegistrant GetRegistrant
        > tGetRegistrant GetRegistrant
        > tGetInstanceOf GetInstanceOf
        > tGetInstanceOf GetInstanceOf
        > tGetPluginType GetPluginType
        > tAdd Add
        > tRemove Remove
        > tContains Contains
        > tContains Contains
        >
        > AddRange
        >
        > RemoveRange
        >
        > Clear
        >
        > GetEnumerator
        >
        > .ctor
        >
        > Count
        >
        > IsSynchronized
        >
        > SyncRoot
        >
        > Item
        >
        > Reset
        >
        > MoveNext
        >
        > .ctor
        >
        > Current
        >
        > GetRegistrants
        >
        > .ctor
        > tRefresh RefreshRegistry
        >
        > GetRegistry
        >
        > SaveRegistry
        >
        > ContainsRegistration
        > tGetPluginTitle GetPluginTitle
        > tGetPluginDescription GetPluginDescription
        >
        > Register
        > tUnregister Unregister
        > tGetPluginObject GetPluginObject
        >
        > .ctor
        > tAbsolutePath GetInstance
        > tGetPluginType GetPluginType
        >
        > LoadAssembly
        >
        > .ctor
        >
        > .ctor
        > tRelativePath FullAssemblyPath
        >
        > Title
        >
        > Description
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        > Now a Code Coverage report (be aware the bars don't appear to
        > be rendering well in the email):
        >
        > Assembly Path :plugin.dll Total Coverage Total IL Instructions
        >
        > 1014/1345 75%
        > ------------------------------
        > Coverage IL Count Function
        >
        > 10/10 100% EF.Plugin.Registrar..cctor
        >
        > 6/6 100% EF.Plugin.Registrar.GetRegistry
        >
        > 11/11 100% EF.Plugin.Registrar.get_Registry
        >
        > 36/36 100% EF.Plugin.Registrar.Reconstitute
        >
        > 11/11 100% EF.Plugin.Registry..ctor
        >
        > 10/10 100% EF.Plugin.Registry.GetEnumerator
        >
        > 10/10 100% EF.Plugin.Registry.GetRegistrant
        >
        > 49/91 53% EF.Plugin.Registry.GetRegistrant
        >
        > 12/12 100% EF.Plugin.Registry.GetInstanceOf
        >
        > 88/92 95% EF.Plugin.Registry.GetInstanceOf
        >
        > 37/76 48% EF.Plugin.Registry.GetPluginType
        >
        > 0/10 0% EF.Plugin.Registry.CopyTo
        >
        > 9/9 100% EF.Plugin.Registry.get_Count
        >
        > 0/6 0% EF.Plugin.Registry.get_IsSynchronized
        >
        > 0/6 0% EF.Plugin.Registry.get_SyncRoot
        >
        > 10/10 100% EF.Plugin.Registry.get_Item
        >
        > 21/21 100% EF.Plugin.Registry.Add
        >
        > 22/28 78% EF.Plugin.Registry.Remove
        >
        > 39/41 95% EF.Plugin.Registry.Contains
        >
        > 9/9 100% EF.Plugin.Registry.Contains
        >
        > 46/57 80% EF.Plugin.Registry.AddRange
        >
        > 50/61 81% EF.Plugin.Registry.RemoveRange
        >
        > 13/13 100% EF.Plugin.Registry.Clear
        >
        > 0/9 0% EF.Plugin.Registry.GetEnumerator
        >
        > 5/5 100% EF.Plugin.Registration..ctor
        >
        > 0/5 0% EF.Plugin.Registrar..ctor
        >
        > 8/8 100% EF.Plugin.Registrar.RefreshRegistry
        >
        > 26/36 72% EF.Plugin.Registrar.SaveRegistry
        >
        > 47/47 100% EF.Plugin.Registrar.ContainsRegistration
        >
        > 12/19 63% EF.Plugin.Registrar.LoadAssembly
        >
        > 28/34 82% EF.Plugin.Registrar.GetPluginTitle
        >
        > 28/32 87% EF.Plugin.Registrar.GetPluginDescription
        >
        > 55/55 100% EF.Plugin.Registrar.Register
        >
        > 55/55 100% EF.Plugin.Registrar.Unregister
        >
        > 22/29 75% EF.Plugin.Registrar.GetPluginObject
        >
        > 6/6 100% EF.Plugin.Registrant..ctor
        >
        > 35/46 76% EF.Plugin.Registrant.get_FullAssemblyPath
        >
        > 0/30 0% EF.Plugin.Registrant.get_Title
        >
        > 0/28 0% EF.Plugin.Registrant.get_Description
        >
        > 0/21 0% EF.Plugin.Registrant..ctor
        >
        > 62/90 68% EF.Plugin.Registrant.GetInstance
        >
        > 50/61 81% EF.Plugin.Registrant.GetPluginType
        >
        > 9/9 100% EF.Plugin.Registrant.LoadAssembly
        >
        > 12/18 66% EF.Plugin.Registrant.LoadAssembly
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        >
        > So by the code coverage reporting 75% coverage I should feel
        > confident about the health of the product, right? Well lets
        > bring the test coverage report of only 39% into the picture.
        > Now I am not so sure. It seems there is a lot of indirect
        > testing. Is that okay? It might be. All by itself there is
        > no problem but it is useful to know if you are trying to
        > figure out why this part of the product is frequinty broken
        > and there is high code coverage, it is all indirect testing.
        > It can also be usful when trying to backfill unit tests.
        > Code coverage is great for helping you get all those branches
        > inside a member but it is a poor tool for determining test
        > coverage of classes and members.
        >
        > What do you think?
        > This is part where you shoot me down and tell me all those
        > things that I did not think of or consider...
        >
        > --
        > Jay Flowers
        > ----------------------------------------------------------------------
        > http://jayflowers.com
        > ---------------------------------------------------------------------
        >
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >
        > To Post a message, send it to: extremeprogramming@...
        >
        > To Unsubscribe, send a blank message to:
        > extremeprogramming-unsubscribe@...
        >
        > ad-free courtesy of objectmentor.com
        > Yahoo! Groups Links
        >
        >
        >
        >
        >
        >
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.