Loading ...
Sorry, an error occurred while loading the content.
 

XP and image processing

Expand Messages
  • Baris Evrim Demiröz
    Hello all! From the first time I saw XP I thought it is a very good methodology. But I have a questions about unit testing. As you may already know, in image
    Message 1 of 11 , Jun 6, 2008
      Hello all!
      From the first time I saw XP I thought it is a very good methodology.
      But I have a questions about unit testing.
      As you may already know, in image processing most of the things work
      with a probability. Let's say you have a method that finds the
      location of faces in a given image, and this algorithm works well 98%
      of the time, and gives false trues (a location that doesn't contain
      any face). How do you test such a method?
    • Kevin
      You would have to find the reasons why it give the false trues and write test conditions for them. If you do not know why it is giving false trues, the unit
      Message 2 of 11 , Jun 6, 2008
        You would have to find the reasons why it give the false trues and write test conditions for them. If you do not know why it is giving false trues, the unit test well not know either.
         
         


        --- On Fri, 6/6/08, Baris Evrim Demiröz <demirozba@...> wrote:

        From: Baris Evrim Demiröz <demirozba@...>
        Subject: [XP] XP and image processing
        To: extremeprogramming@yahoogroups.com
        Date: Friday, June 6, 2008, 1:50 PM






        Hello all!
        From the first time I saw XP I thought it is a very good methodology.
        But I have a questions about unit testing.
        As you may already know, in image processing most of the things work
        with a probability. Let's say you have a method that finds the
        location of faces in a given image, and this algorithm works well 98%
        of the time, and gives false trues (a location that doesn't contain
        any face). How do you test such a method?


















        [Non-text portions of this message have been removed]
      • D. André Dhondt
        This sounds like something too complicated for a unit test. I imagine I d have UTs for smaller tasks, like finding dark contiguous regions, or an edge, or
        Message 3 of 11 , Jun 6, 2008
          This sounds like something too complicated for a unit test. I imagine I'd
          have UTs for smaller tasks, like finding dark contiguous regions, or an
          edge, or other more clearly defined small tasks. Then I'd have Acceptance
          Tests with images that had been interpreted by a human, and upon running a
          suite of tests against a series of different face recognition algorithms
          you'd get feedback on how good that overall algorithm is.

          On Fri, Jun 6, 2008 at 1:50 PM, Baris Evrim Demiröz <demirozba@...>
          wrote:

          > Hello all!
          > From the first time I saw XP I thought it is a very good methodology.
          > But I have a questions about unit testing.
          > As you may already know, in image processing most of the things work
          > with a probability. Let's say you have a method that finds the
          > location of faces in a given image, and this algorithm works well 98%
          > of the time, and gives false trues (a location that doesn't contain
          > any face). How do you test such a method?
          >
          >
          >



          --
          D. André Dhondt
          mobile: 267-283-8270
          home: 267-286-6875

          If you're a software developer in the area, join Agile Philly (
          http://groups.yahoo.com/group/agilephilly/)!


          [Non-text portions of this message have been removed]
        • Kevin
          I think you could unit test it by using images that you know the information about. The ones that work should return the correct information, the ones that
          Message 4 of 11 , Jun 6, 2008
            I think you could unit test it by using images that you know the information about. The ones that work should return the correct information, the ones that knowenly don't work should return the correct information that says it cant find them.

            --- On Fri, 6/6/08, D. André Dhondt <d.andre.dhondt@...> wrote:

            From: D. André Dhondt <d.andre.dhondt@...>
            Subject: Re: [XP] XP and image processing
            To: extremeprogramming@yahoogroups.com
            Date: Friday, June 6, 2008, 3:52 PM






            This sounds like something too complicated for a unit test. I imagine I'd
            have UTs for smaller tasks, like finding dark contiguous regions, or an
            edge, or other more clearly defined small tasks. Then I'd have Acceptance
            Tests with images that had been interpreted by a human, and upon running a
            suite of tests against a series of different face recognition algorithms
            you'd get feedback on how good that overall algorithm is.

            On Fri, Jun 6, 2008 at 1:50 PM, Baris Evrim Demiröz <demirozba@yahoo. com>
            wrote:

            > Hello all!
            > From the first time I saw XP I thought it is a very good methodology.
            > But I have a questions about unit testing.
            > As you may already know, in image processing most of the things work
            > with a probability. Let's say you have a method that finds the
            > location of faces in a given image, and this algorithm works well 98%
            > of the time, and gives false trues (a location that doesn't contain
            > any face). How do you test such a method?
            >
            >
            >

            --
            D. André Dhondt
            mobile: 267-283-8270
            home: 267-286-6875

            If you're a software developer in the area, join Agile Philly (
            http://groups. yahoo.com/ group/agilephill y/)!

            [Non-text portions of this message have been removed]


















            [Non-text portions of this message have been removed]
          • Phlip
            [CCd to testfirstuserinterfaces@yahoogroups.com ] ... Study the source to Cairo (part of Gtk+). It uses a system called golden master test . Each test case
            Message 5 of 11 , Jun 6, 2008
              [CCd to testfirstuserinterfaces@yahoogroups.com ]

              Baris Evrim Demiröz wrote:

              > From the first time I saw XP I thought it is a very good methodology.
              > But I have a questions about unit testing.
              > As you may already know, in image processing most of the things work
              > with a probability. Let's say you have a method that finds the
              > location of faces in a given image, and this algorithm works well 98%
              > of the time, and gives false trues (a location that doesn't contain
              > any face). How do you test such a method?

              Study the source to Cairo (part of Gtk+). It uses a system called "golden master
              test". Each test case (at this level) draws a little graphic, such as one
              diagonal line. Then the test case byte-compares the image to a png file.

              If the test output is right and the png file is wrong, engineers replace the png
              file with the new output. That's why the master files are "golden" - they are
              part of your versioned source.

              This system won't prove you match faces. TDD cannot test that crypto makes your
              app bulletproof, or that compression absolutely minimizes your files, or that a
              Turing loop will never halt.

              This system should stop you if you make a change too big for its tolerances.
              This frees your edits up to make whatever change you like. Get a battery of
              known faces, and test that they all get recognized to within their expected
              variations.

              When you actually TDD, you are simply matching tests to low-level code, to check
              it manipulates its bits correctly. Most of your tests should run below the level
              of the faces.

              --
              Phlip
            • Carfield Yim
              Never tried test in this area, but I see some article about this, may be you can take a look? http://www.xprogramming.com/xpmag/dbcShotgunGray.htm
              Message 6 of 11 , Jun 6, 2008
                Never tried test in this area, but I see some article about this, may
                be you can take a look?

                http://www.xprogramming.com/xpmag/dbcShotgunGray.htm
                http://www.xprogramming.com/xpmag/dbcShootingThePicture.htm
                http://www.xprogramming.com/xpmag/dbcShootingForEffect.htm
                http://www.xprogramming.com/xpmag/dbcImprovingDrawingCode.htm
                http://www.xprogramming.com/xpmag/dbcShootingInCircles.htm

                On Sat, Jun 7, 2008 at 1:50 AM, Baris Evrim Demiröz <demirozba@...> wrote:
                > Hello all!
                > From the first time I saw XP I thought it is a very good methodology.
                > But I have a questions about unit testing.
                > As you may already know, in image processing most of the things work
                > with a probability. Let's say you have a method that finds the
                > location of faces in a given image, and this algorithm works well 98%
                > of the time, and gives false trues (a location that doesn't contain
                > any face). How do you test such a method?
                >
                >
                > ------------------------------------
                >
                > To Post a message, send it to: extremeprogramming@...
                >
                > To Unsubscribe, send a blank message to: extremeprogramming-unsubscribe@...
                >
                > ad-free courtesy of objectmentor.comYahoo! Groups Links
                >
                >
                >
                >
              • John Roth
                While image processing is difficult, what you re looking for isn t a unit test - it s an acceptance test. As Phlip says, it simply isn t low enough level for a
                Message 7 of 11 , Jun 6, 2008
                  While image processing is difficult, what you're
                  looking for isn't a unit test - it's an acceptance
                  test. As Phlip says, it simply isn't low enough level
                  for a unit test.

                  Since it's an acceptance test with poorly defined
                  criteria, you throw a boatload of images at it, each of which
                  has a known output (the "Guru says" pattern).
                  Then you inspect the differences to see whether
                  they fall into known problem areas (shadows, for
                  example in your case).

                  The advantages of this approach are:

                  1. You've got an automated regression test suite
                  you can qualify algorithm improvements against.

                  2. You can add test files indefinitely, qualifying each
                  suite to avoid selection bias, e.g. "all the advertisements
                  from last month's Vogue".

                  John Roth



                  ----- Original Message -----
                  From: "Baris Evrim Demiröz" <demirozba@...>
                  To: <extremeprogramming@yahoogroups.com>
                  Sent: Friday, June 06, 2008 11:50 AM
                  Subject: [XP] XP and image processing


                  Hello all!
                  From the first time I saw XP I thought it is a very good methodology.
                  But I have a questions about unit testing.
                  As you may already know, in image processing most of the things work
                  with a probability. Let's say you have a method that finds the
                  location of faces in a given image, and this algorithm works well 98%
                  of the time, and gives false trues (a location that doesn't contain
                  any face). How do you test such a method?
                • Baris Evrim Demiröz
                  Thanks for all the replies. I will have a look at the things you all have pointed. The conclusion I have drawn is this task is too complicated for unit
                  Message 8 of 11 , Jun 7, 2008
                    Thanks for all the replies. I will have a look at the things you all
                    have pointed.
                    The conclusion I have drawn is this task is too complicated for unit
                    testing. Instead of unit testing face locating method, I should write
                    tests for smaller tasks like image resizing. And those test should
                    check for the image width/height, handling Null cases etc.
                    These are my opinions before looking to the links you people have
                    given, after reading they may change ;)
                    Thanks again!

                    --- In extremeprogramming@yahoogroups.com, John Roth <JohnRoth1@...>
                    wrote:
                    >
                    > While image processing is difficult, what you're
                    > looking for isn't a unit test - it's an acceptance
                    > test. As Phlip says, it simply isn't low enough level
                    > for a unit test.
                    >
                    > Since it's an acceptance test with poorly defined
                    > criteria, you throw a boatload of images at it, each of which
                    > has a known output (the "Guru says" pattern).
                    > Then you inspect the differences to see whether
                    > they fall into known problem areas (shadows, for
                    > example in your case).
                    >
                    > The advantages of this approach are:
                    >
                    > 1. You've got an automated regression test suite
                    > you can qualify algorithm improvements against.
                    >
                    > 2. You can add test files indefinitely, qualifying each
                    > suite to avoid selection bias, e.g. "all the advertisements
                    > from last month's Vogue".
                    >
                    > John Roth
                    >
                    >
                    >
                    > ----- Original Message -----
                    > From: "Baris Evrim Demiröz" <demirozba@...>
                    > To: <extremeprogramming@yahoogroups.com>
                    > Sent: Friday, June 06, 2008 11:50 AM
                    > Subject: [XP] XP and image processing
                    >
                    >
                    > Hello all!
                    > From the first time I saw XP I thought it is a very good methodology.
                    > But I have a questions about unit testing.
                    > As you may already know, in image processing most of the things work
                    > with a probability. Let's say you have a method that finds the
                    > location of faces in a given image, and this algorithm works well 98%
                    > of the time, and gives false trues (a location that doesn't contain
                    > any face). How do you test such a method?
                    >
                  • Ron Jeffries
                    ... What do we want to know about the method? If we have a given saved pictures, do we want it to get the same answer all the time? What if we had 100 pictures
                    Message 9 of 11 , Jun 7, 2008
                      Hello, Baris. On Friday, June 6, 2008, at 1:50:37 PM, you wrote:

                      >>From the first time I saw XP I thought it is a very good methodology.
                      > But I have a questions about unit testing.
                      > As you may already know, in image processing most of the things work
                      > with a probability. Let's say you have a method that finds the
                      > location of faces in a given image, and this algorithm works well 98%
                      > of the time, and gives false trues (a location that doesn't contain
                      > any face). How do you test such a method?

                      What do we want to know about the method? If we have a given saved
                      pictures, do we want it to get the same answer all the time?

                      What if we had 100 pictures and it correctly classified 95 of them?
                      What if it did 99?

                      Ron Jeffries
                      www.XProgramming.com
                      Inigo Montoya: You are wonderful!
                      Man in Black: Thank you. I have worked hard to become so.
                    • Chris Wheeler
                      Yes, good articles. Ron and Chet wrote those in response to this exact type of issue, if memory serves.... Specifically, there was a thread around things that
                      Message 10 of 11 , Jun 7, 2008
                        Yes, good articles. Ron and Chet wrote those in response to this exact type
                        of issue, if memory serves.... Specifically, there was a thread around
                        things that couldn't be TDD'd, and those guys do a decent job of answering
                        the question 'Can graphical stuff get developed using TDD?'

                        Chris.

                        On Fri, Jun 6, 2008 at 11:46 PM, Carfield Yim <carfield@...> wrote:

                        > Never tried test in this area, but I see some article about this, may
                        > be you can take a look?
                        >
                        > http://www.xprogramming.com/xpmag/dbcShotgunGray.htm
                        > http://www.xprogramming.com/xpmag/dbcShootingThePicture.htm
                        > http://www.xprogramming.com/xpmag/dbcShootingForEffect.htm
                        > http://www.xprogramming.com/xpmag/dbcImprovingDrawingCode.htm
                        > http://www.xprogramming.com/xpmag/dbcShootingInCircles.htm
                        >
                        > On Sat, Jun 7, 2008 at 1:50 AM, Baris Evrim Demiröz <demirozba@...>
                        > wrote:
                        > > Hello all!
                        > > From the first time I saw XP I thought it is a very good methodology.
                        > > But I have a questions about unit testing.
                        > > As you may already know, in image processing most of the things work
                        > > with a probability. Let's say you have a method that finds the
                        > > location of faces in a given image, and this algorithm works well 98%
                        > > of the time, and gives false trues (a location that doesn't contain
                        > > any face). How do you test such a method?
                        > >
                        > >
                        > > ------------------------------------
                        > >
                        > > To Post a message, send it to: extremeprogramming@...
                        > >
                        > > To Unsubscribe, send a blank message to:
                        > extremeprogramming-unsubscribe@...
                        > >
                        > > ad-free courtesy of objectmentor.comYahoo! Groups Links
                        > >
                        > >
                        > >
                        > >
                        >
                        > ------------------------------------
                        >
                        > To Post a message, send it to: extremeprogramming@...
                        >
                        > To Unsubscribe, send a blank message to:
                        > extremeprogramming-unsubscribe@...
                        >
                        > ad-free courtesy of objectmentor.comYahoo! Groups Links
                        >
                        >
                        >
                        >


                        [Non-text portions of this message have been removed]
                      • Phlip
                        ... 3. brute-force acceptance tests make a good-enough regression detector that you can now start refactoring the low-level code to add its unit tests.
                        Message 11 of 11 , Jun 7, 2008
                          > 1. You've got an automated regression test suite
                          > you can qualify algorithm improvements against.
                          >
                          > 2. You can add test files indefinitely, qualifying each
                          > suite to avoid selection bias, e.g. "all the advertisements
                          > from last month's Vogue".

                          3. brute-force acceptance tests make a good-enough regression detector that you
                          can now start refactoring the low-level code to add its unit tests.
                        Your message has been successfully submitted and would be delivered to recipients shortly.