Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Mock object for testing email

Expand Messages
  • Marcus Baker
    Hi... ... Sorry to piggyback your thread a bit late Steve, but the list is too busy to track down your earlier conversations. I have never satisfactorily got
    Message 1 of 9 , Feb 26, 2005
    • 0 Attachment
      Hi...

      Steve Brennan wrote:
      > Today I implemented a mock object that has the same interface as the class
      > that sends mail for our system.

      Sorry to piggyback your thread a bit late Steve, but the list is too
      busy to track down your earlier conversations.

      I have never satisfactorily got e-mail testing sorted either, especially
      for deployment and acceptance testing. Would it be possible for people
      to add their solutions to this mini-survey along with their preferences...?

      1) DI/Registry to substitute a mock/fake. It means being able to
      effectively select a class in your configuration files. This can mean
      restarting your app. after the tests to reconfigure it. The trouble is
      what are you testing when you do this? Of course we do this for unit tests.

      2) Creating a fake mail transfer agent to capture traffic. The
      configuration only needs a port and host to switch during testing. Your
      outgoing mail class has to be able to change the SMTP port unless the
      test takes down the local MTA temporarily.

      3) Setting up a fake reciever/POP account and reading that (yuk). Too
      many dependencies across the network and too much setting up to do on
      development boxes.

      4) Reading the mail queue on a disabled MTA. For technical reasons and a
      general lack of understanding, I could not get anywhere with this.
      Anyway it just seems far too fiddly.

      I am currently on number 2 (the fakemail project on SF). Any thoughts?

      >
      > --Steve Brennan

      yours, Marcus
      --
      Marcus Baker, marcus@... - http://www.lastcraft.com/
      PHP London every first Thursday - http://www.phplondon.org/
    • Alain Ravet
      Marcus, I guess my solution is 2B/ using a dedicated MTA: I once used James to handle the emails send/read by my tests. Cons: - James has to be running before
      Message 2 of 9 , Feb 27, 2005
      • 0 Attachment
        Marcus,

        I guess my solution is
        2B/ using a dedicated MTA:

        I once used James to handle the emails send/read by my tests.
        Cons:
        - James has to be running before you test

        The setup requires creating a new user, with an empty account +
        deleting. I can't remember how slow/fast it was.

        Alain
      • William Pietri
        ... My inclination is to combine several of the things you mention. For unit testing an in-system mail sender, I use a fake MTA. It s an ultra-simple SMTP
        Message 3 of 9 , Feb 27, 2005
        • 0 Attachment
          On Sat, 2005-02-26 at 20:40 +0000, Marcus Baker wrote:
          > I have never satisfactorily got e-mail testing sorted either,
          > especially for deployment and acceptance testing. Would it be possible
          > for people to add their solutions to this mini-survey along with their
          > preferences...?

          My inclination is to combine several of the things you mention.

          For unit testing an in-system mail sender, I use a fake MTA. It's an
          ultra-simple SMTP server that just knows the minimum necessary to accept
          mail, with an API for examining the messages it receives. Because it's a
          unit test, it's easy to tell the production code to listen on a non-
          standard port.

          For unit testing the parts of the system that send messages, I'll either
          use a mock of the mail sending service or I'll set the mail sending
          service up in a way where it just queues messages and then I'll examine
          the queue.

          And then for acceptance testing the whole system, I've done it a couple
          of ways. Using the fake MTA is easiest, but it does involve running the
          production system with different preferences than the live system.

          One one project, that was unsatisfactory, so we instead rigged the mail
          server to deliver any mail sent to example.com to a special POP mailbox.
          Each test tagged the emails it triggered with a unique ID, and then we
          wrote a special POP client that would pull out only the messages you
          were interested in.


          The fake MTA is certainly easiest; that's under a day's work for
          somebody who knows SMTP. (For those who don't, ethereal is your friend.)
          Mocking out the mail sending service can be easy or hard depending on
          your architecture. And the full-round-trip-via-POP approach is a bit of
          work to get working happily in a multi-developer environment.


          Is that the sort of information you were looking for?


          Regards,

          William


          --
          William Pietri <william@...>
        • Marcus Baker
          Hi... ... Well, it s confirmation that we are both going about it the same way. I have a great reluctance to put test data into the live system though (it
          Message 4 of 9 , Feb 27, 2005
          • 0 Attachment
            Hi...

            William Pietri wrote:
            > Is that the sort of information you were looking for?

            Well, it's confirmation that we are both going about it the same way. I
            have a great reluctance to put test data into the live system though (it
            tends to outlive the test).

            > Regards,
            >
            > William

            yours, Marcus
            --
            Marcus Baker, marcus@... - http://www.lastcraft.com/
            PHP London every first Thursday - http://www.phplondon.org/
          • William Pietri
            ... Ah, in my case I always do my acceptance testing on a copy of the production system, so that s not a worry for me. I take it there s a reason you haven t
            Message 5 of 9 , Feb 27, 2005
            • 0 Attachment
              On Sun, 2005-02-27 at 20:33 +0000, Marcus Baker wrote:
              > Well, it's confirmation that we are both going about it the same way.
              > I have a great reluctance to put test data into the live system though
              > (it tends to outlive the test).

              Ah, in my case I always do my acceptance testing on a copy of the
              production system, so that's not a worry for me. I take it there's a
              reason you haven't done something similar?

              William

              --
              William Pietri <william@...>
            • Marcus Baker
              Hi... ... Ooh, feel those barbs...;) We have a clone database on the live servers for testing during roll-outs, but we don t run the tests with the real live
              Message 6 of 9 , Feb 27, 2005
              • 0 Attachment
                Hi...

                William Pietri wrote:
                > Ah, in my case I always do my acceptance testing on a copy of the
                > production system, so that's not a worry for me. I take it there's a
                > reason you haven't done something similar?

                Ooh, feel those barbs...;)

                We have a clone database on the live servers for testing during
                roll-outs, but we don't run the tests with the "real" live data in case
                of accidents. I minute of downtime equates to about 5 support tickets.
                This means we add it to the clone and pay for it in configuration file
                twiddling in the roll-out process.

                We did have a POP reader one before, but we like to run the same tests
                on the in office integration machines and the dev. boxes. This was
                causing way too much pain in the office and is now being replaced with
                the fake MTA. As you say, it's about a day's work counting the
                inevitable "I thought we knew how this worked" ethereal debugging session.

                > William

                yours, Marcus
                --
                Marcus Baker, marcus@... - http://www.lastcraft.com/
                PHP London every first Thursday - http://www.phplondon.org/
              • Steven J. Owens
                ... I feel the same way. But a while back I made a command decision to simply include a test user in every one of our deployments. Instead of trying to hide
                Message 7 of 9 , Apr 1, 2005
                • 0 Attachment
                  On Sun, Feb 27, 2005 at 08:33:39PM +0000, Marcus Baker wrote:
                  > William Pietri wrote:
                  > > Is that the sort of information you were looking for?
                  >
                  > Well, it's confirmation that we are both going about it the same way. I
                  > have a great reluctance to put test data into the live system though (it
                  > tends to outlive the test).

                  I feel the same way. But a while back I made a command decision
                  to simply include a test user in every one of our deployments.
                  Instead of trying to hide the testing process from the customer, why
                  not enlist the customer in the testing process?

                  We're using an ASP business model - the software is a webapp, it
                  all lives on our servers and our customers log into it over the web.
                  (By the by, I read a couple months ago that the hot new buzzword is
                  BPO, "Business Process Outsourcing", but yesterday I learned an even
                  more bleeding-edge term, SAAS, "Software As A Service".

                  One up side of this is that we can (and do) add new features
                  often, sometimes more than once a month. Sometimes these are general
                  features, added to all our deployments, sometimes they're specific
                  configuration tweaks that a customer requests. So far I've been
                  assiduously side-stepping the problem of having different code-bases,
                  by making all behavioral variation controlled by configuration
                  variables (sometimes taking the "clever brute force" approach of
                  having different classes and a configuration variable that determines
                  which one gets invoked).

                  Our system is complex enough that we sometimes really need to do
                  an end-to-end test. Part of that is because we interface with
                  external systems and I have not (yet!) implemented a fake stub server
                  to simulate the external system. Part of it is that testing the
                  external system is a key problem, and the technology in question is a
                  big pain. So even though we test the heck out of changes before
                  deploying them, we also include two test users in every deployment,
                  and we train the customers to react to them appropriately.

                  This is even a useful tool for us. It lets us give the customer
                  some visibility into the testing. When we get a problem report, we
                  can quickly run a test with the test user and the customer _sees_ that
                  we're responding to their report.

                  --
                  Steven J. Owens
                  puff@...

                  "I'm going to make broad, sweeping generalizations and strong,
                  declarative statements, because otherwise I'll be here all night and
                  this document will be four times longer and much less fun to read.
                  Take it all with a grain of salt." - http://darksleep.com/notablog
                • Jeff Grigg
                  ... Mine too. Several years ago, working on a small business app that emailed data between laptops and a central system (one for each customer), we had to
                  Message 8 of 9 , Apr 2, 2005
                  • 0 Attachment
                    > --- Marcus Baker wrote:
                    >> I have never satisfactorily got e-mail testing sorted either,
                    >> especially for deployment and acceptance testing. Would it be
                    >> possible for people to add their solutions to this mini-survey
                    >> along with their preferences...?

                    --- William Pietri <william@s...> wrote:
                    > My inclination is to combine several of the things you mention.

                    Mine too.

                    Several years ago, working on a small business app that emailed data
                    between laptops and a central system (one for each customer), we had
                    to interface to a variety of email systems through Simple MAPI and a
                    few other mail APIs. (Remarkable note: No one, not even Microsoft,
                    actually conformed to the S-MAPI interface, as described in
                    Microsoft's documentation of it. Go figure. ;-)

                    So for acceptance testing, we'd install the actual mail system under
                    test and run with it. Not with mocks or stubs on either end, but
                    with the real business system sending real messages through the real
                    email system and interpreting them. Pretty good acceptance test,
                    but time-consuming.

                    For more local tests, we used our simplest mail provider --
                    essentially a NullObject implementation that could actually
                    accomplish the file transfer -- if the systems were carefully
                    configured to use shared directories. For most testing, this worked
                    well for us, as the code to communicate with each specific mail
                    system rarely changed, once it was working properly, but the
                    business and networking logic on our side of a generic mail API of
                    our design was constantly changing. So this focused our testing on
                    the code that was changing -- which is where the testing was most
                    needed. ;->
                  • Marcus Baker
                    Hi... ... To save everyone else that day s work... http://www.lastcraft.com/fakemail.php yours, Marcus -- Marcus Baker, marcus@lastcraft.com -
                    Message 9 of 9 , May 6, 2005
                    • 0 Attachment
                      Hi...

                      William Pietri wrote:
                      > The fake MTA is certainly easiest; that's under a day's work for
                      > somebody who knows SMTP. (For those who don't, ethereal is your friend.)

                      To save everyone else that day's work...
                      http://www.lastcraft.com/fakemail.php

                      yours, Marcus
                      --
                      Marcus Baker, marcus@... - http://www.lastcraft.com/
                      PHP London every first Thursday - http://www.phplondon.org/
                    Your message has been successfully submitted and would be delivered to recipients shortly.