Loading ...
Sorry, an error occurred while loading the content.

Re: A question about Milters and Thread Safety

Expand Messages
  • Anton Yuzhaninov
    ... Yes, it must be thread-safe. You can write milters using sendmail s libmilter, which may be used only for creating threaded milters. Theoretically possible
    Message 1 of 15 , Sep 3, 2007
    • 0 Attachment
      On 03.09.2007 22:17, Ronald F. Guilmette wrote:
      > Anyway, recently I got to wondering: Do milter programs that are
      > only going to be used in conjunction with Postfix (and _not_ any
      > other kind of mail server, e.g. Sendmail) really need to be thread-
      > safe?

      Yes, it must be thread-safe.

      You can write milters using sendmail's libmilter, which may be used only for creating
      threaded milters.

      Theoretically possible to write other library for writing milters (without threads), but I
      don't know other libraries except libmilter.

      --
      WBR,
      Anton Yuzhaninov
    • Ronald F. Guilmette
      In message , ... Yes, that s what I thought it meant. Ugg! I hate to have to do all of this extra work, just
      Message 2 of 15 , Sep 3, 2007
      • 0 Attachment
        In message <20070903182725.458B91F3E9D@...>,
        wietse@... (Wietse Venema) wrote:

        >Ronald F. Guilmette:
        >> Anyway, recently I got to wondering: Do milter programs that are
        >> only going to be used in conjunction with Postfix (and _not_ any
        >> other kind of mail server, e.g. Sendmail) really need to be thread-
        >> safe?
        >
        >Yes, it needs to be thread safe. This means jumping hoops when
        >you maintain persistent data structures, global variables. It
        >also means you can't call functions that aren't thread safe.

        Yes, that's what I thought it meant.

        Ugg! I hate to have to do all of this extra work, just to write a
        rather simple milter.

        Just for the sake of my edification, could you please also tell me
        _why_ a Postfix milter has to be thread safe? I mean Postfix's smtpd
        processes don't actually _use_ separate threads, do they? So where
        does the requirement come from? What would fail if I wrote a non-
        thread-safe milter and tried to use it with Postfix? Would my data
        structures that are updated/modified in a non-thread-safe way really
        become corrupted? If so, how would that happen... even in the total
        absence of multiple threads?


        Signed,
        Perplexed
      • Wietse Venema
        ... There is no such thing as a Postfix Milter. Your milter application runs on top of libmilter. Wietse
        Message 3 of 15 , Sep 3, 2007
        • 0 Attachment
          Ronald F. Guilmette:
          >
          > In message <20070903182725.458B91F3E9D@...>,
          > wietse@... (Wietse Venema) wrote:
          >
          > >Ronald F. Guilmette:
          > >> Anyway, recently I got to wondering: Do milter programs that are
          > >> only going to be used in conjunction with Postfix (and _not_ any
          > >> other kind of mail server, e.g. Sendmail) really need to be thread-
          > >> safe?
          > >
          > >Yes, it needs to be thread safe. This means jumping hoops when
          > >you maintain persistent data structures, global variables. It
          > >also means you can't call functions that aren't thread safe.
          >
          > Yes, that's what I thought it meant.
          >
          > Ugg! I hate to have to do all of this extra work, just to write a
          > rather simple milter.
          >
          > Just for the sake of my edification, could you please also tell me
          > _why_ a Postfix milter has to be thread safe?

          There is no such thing as a Postfix Milter.

          Your milter application runs on top of libmilter.

          Wietse
        • Ronald F. Guilmette
          In message , ... Well, that sort-of begs the question... What if it _didn t_ run on top of libmilter? What if
          Message 4 of 15 , Sep 3, 2007
          • 0 Attachment
            In message <20070903193047.4AF311F3EA1@...>,
            wietse@... (Wietse Venema) wrote:

            >Ronald F. Guilmette:
            >> Just for the sake of my edification, could you please also tell me
            >> _why_ a Postfix milter has to be thread safe?
            >
            >There is no such thing as a Postfix Milter.
            >
            >Your milter application runs on top of libmilter.

            Well, that sort-of begs the question... What if it _didn't_ run on
            top of libmilter? What if it skipped that and tried to interface
            to Postfix smtpd directly? Would my milter still have to be
            thread-safe under that scenario?

            (Please forgive me for making such an issue of this, but it really
            would be a major pain in the ass to make what I have in mind thread-
            safe. A significant part of the problem with doing that... but
            certainly not the only problem... is that the spam filter that I'd
            like to turn into a milter does DNS lookups. Sadly, I am not aware
            of any thread-safe DNS primitives library, so I kind-of think that
            I'd be stuck trying to write my own, from scratch. Am I just looking
            in the Wrong Places? Does anybody here happen to know of a reasonably
            priced thread-safe DNS primitives library?)


            Regards,
            rfg
          • Wietse Venema
            ... Good luck. You will need it, and I won t be available to answer questions. Wietse
            Message 5 of 15 , Sep 3, 2007
            • 0 Attachment
              Ronald F. Guilmette:
              >
              > In message <20070903193047.4AF311F3EA1@...>,
              > wietse@... (Wietse Venema) wrote:
              >
              > >Ronald F. Guilmette:
              > >> Just for the sake of my edification, could you please also tell me
              > >> _why_ a Postfix milter has to be thread safe?
              > >
              > >There is no such thing as a Postfix Milter.
              > >
              > >Your milter application runs on top of libmilter.
              >
              > Well, that sort-of begs the question... What if it _didn't_ run on
              > top of libmilter?

              Good luck. You will need it, and I won't be available to answer
              questions.

              Wietse
            • steeeeeveee@gmx.net
              ... What spam filter is that? ... -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN:
              Message 6 of 15 , Sep 3, 2007
              • 0 Attachment
                -------- Original-Nachricht --------
                > Datum: Mon, 03 Sep 2007 12:58:15 -0700
                > Von: "Ronald F. Guilmette" <rfg@...>
                > An: postfix-users@...
                > Betreff: Re: A question about Milters and Thread Safety

                >
                > In message <20070903193047.4AF311F3EA1@...>,
                > wietse@... (Wietse Venema) wrote:
                >
                > >Ronald F. Guilmette:
                > >> Just for the sake of my edification, could you please also tell me
                > >> _why_ a Postfix milter has to be thread safe?
                > >
                > >There is no such thing as a Postfix Milter.
                > >
                > >Your milter application runs on top of libmilter.
                >
                > Well, that sort-of begs the question... What if it _didn't_ run on
                > top of libmilter? What if it skipped that and tried to interface
                > to Postfix smtpd directly? Would my milter still have to be
                > thread-safe under that scenario?
                >
                > (Please forgive me for making such an issue of this, but it really
                > would be a major pain in the ass to make what I have in mind thread-
                > safe. A significant part of the problem with doing that... but
                > certainly not the only problem... is that the spam filter that I'd
                > like to turn into a milter does DNS lookups.
                >
                What spam filter is that?


                > Sadly, I am not aware
                > of any thread-safe DNS primitives library, so I kind-of think that
                > I'd be stuck trying to write my own, from scratch. Am I just looking
                > in the Wrong Places? Does anybody here happen to know of a reasonably
                > priced thread-safe DNS primitives library?)
                >
                >
                > Regards,
                > rfg

                --
                Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
                Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer
              • Michael Tokarev
                ... [] ... Your app can implement/talk milter protocol internally. You don t interface to Postfix smtpd directly, but you use sockets to talk to Postfix.
                Message 7 of 15 , Sep 3, 2007
                • 0 Attachment
                  Ronald F. Guilmette wrote:
                  > In message <20070903193047.4AF311F3EA1@...>,
                  > wietse@... (Wietse Venema) wrote:
                  []
                  >> Your milter application runs on top of libmilter.
                  >
                  > Well, that sort-of begs the question... What if it _didn't_ run on
                  > top of libmilter? What if it skipped that and tried to interface
                  > to Postfix smtpd directly? Would my milter still have to be
                  > thread-safe under that scenario?

                  Your app can implement/talk milter protocol internally. You don't
                  interface to Postfix smtpd directly, but you use sockets to talk
                  to Postfix. Speaking of the protocol - well, one (reference)
                  implementation of server-side milter (Postfix is on client side
                  in this case) is libmilter, which use threads - you can roll your
                  own...

                  > (Please forgive me for making such an issue of this, but it really
                  > would be a major pain in the ass to make what I have in mind thread-
                  > safe. A significant part of the problem with doing that... but
                  > certainly not the only problem... is that the spam filter that I'd
                  > like to turn into a milter does DNS lookups. Sadly, I am not aware
                  > of any thread-safe DNS primitives library, so I kind-of think that
                  > I'd be stuck trying to write my own, from scratch. Am I just looking
                  > in the Wrong Places? Does anybody here happen to know of a reasonably
                  > priced thread-safe DNS primitives library?)

                  Ever looked at my udns -- http://www.corpit.ru/mjt/udns.html ?

                  /mjt
                • Victor Duchovni
                  ... No, but it needs to handle concurrent requests for service in some fashion, it could be forking or preforking, ... ... If you don t use libmilter, and have
                  Message 8 of 15 , Sep 4, 2007
                  • 0 Attachment
                    On Mon, Sep 03, 2007 at 12:58:15PM -0700, Ronald F. Guilmette wrote:

                    >
                    > In message <20070903193047.4AF311F3EA1@...>,
                    > wietse@... (Wietse Venema) wrote:
                    >
                    > >Ronald F. Guilmette:
                    > >> Just for the sake of my edification, could you please also tell me
                    > >> _why_ a Postfix milter has to be thread safe?
                    > >
                    > >There is no such thing as a Postfix Milter.
                    > >
                    > >Your milter application runs on top of libmilter.
                    >
                    > Well, that sort-of begs the question... What if it _didn't_ run on
                    > top of libmilter? What if it skipped that and tried to interface
                    > to Postfix smtpd directly? Would my milter still have to be
                    > thread-safe under that scenario?

                    No, but it needs to handle concurrent requests for service in some
                    fashion, it could be forking or preforking, ...

                    > (Please forgive me for making such an issue of this, but it really
                    > would be a major pain in the ass to make what I have in mind thread-
                    > safe. A significant part of the problem with doing that... but
                    > certainly not the only problem... is that the spam filter that I'd
                    > like to turn into a milter does DNS lookups. Sadly, I am not aware
                    > of any thread-safe DNS primitives library, so I kind-of think that
                    > I'd be stuck trying to write my own, from scratch. Am I just looking
                    > in the Wrong Places? Does anybody here happen to know of a reasonably
                    > priced thread-safe DNS primitives library?)

                    If you don't use libmilter, and have some way of handling concurrent
                    requests without blocking for very long, you can use a non-threaded
                    milter implementation.

                    --
                    Viktor.

                    Disclaimer: off-list followups get on-list replies or get ignored.
                    Please do not ignore the "Reply-To" header.

                    To unsubscribe from the postfix-users list, visit
                    http://www.postfix.org/lists.html or click the link below:
                    <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                    If my response solves your problem, the best way to thank me is to not
                    send an "it worked, thanks" follow-up. If you must respond, please put
                    "It worked, thanks" in the "Subject" so I can delete these quickly.
                  • Ronald F. Guilmette
                    In message , ... Thanks. That s exactly what I suspected... and I may indeed have to do that. ... Funny you should mention
                    Message 9 of 15 , Sep 4, 2007
                    • 0 Attachment
                      In message <46DC7A95.8040503@...>,
                      Michael Tokarev <mjt@...> wrote:

                      >Ronald F. Guilmette wrote:
                      >> In message <20070903193047.4AF311F3EA1@...>,
                      >> wietse@... (Wietse Venema) wrote:
                      >[]
                      >>> Your milter application runs on top of libmilter.
                      >>
                      >> Well, that sort-of begs the question... What if it _didn't_ run on
                      >> top of libmilter? What if it skipped that and tried to interface
                      >> to Postfix smtpd directly? Would my milter still have to be
                      >> thread-safe under that scenario?
                      >
                      >Your app can implement/talk milter protocol internally. You don't
                      >interface to Postfix smtpd directly, but you use sockets to talk
                      >to Postfix. Speaking of the protocol - well, one (reference)
                      >implementation of server-side milter (Postfix is on client side
                      >in this case) is libmilter, which use threads - you can roll your
                      >own...

                      Thanks. That's exactly what I suspected... and I may indeed have to do
                      that.

                      >> (Please forgive me for making such an issue of this, but it really
                      >> would be a major pain in the ass to make what I have in mind thread-
                      >> safe. A significant part of the problem with doing that... but
                      >> certainly not the only problem... is that the spam filter that I'd
                      >> like to turn into a milter does DNS lookups. Sadly, I am not aware
                      >> of any thread-safe DNS primitives library, so I kind-of think that
                      >> I'd be stuck trying to write my own, from scratch. Am I just looking
                      >> in the Wrong Places? Does anybody here happen to know of a reasonably
                      >> priced thread-safe DNS primitives library?)
                      >
                      >Ever looked at my udns -- http://www.corpit.ru/mjt/udns.html ?

                      Funny you should mention that! I was looking at that just yesterday.

                      That _may_ be the solution to my problem, but just as a general matter,
                      I would greatly prefer not to have to worry about thread safety at all
                      if I can avoid it. Also, programming in an event driven style is likely
                      to be a good deal harder than just making simple calls to res_query()...
                      as my filter does now... and then just waiting, synchronously, for the
                      result to come back.


                      Regards,
                      rfg
                    • mouss
                      ... are you confusing thread-safe with asynchronous? from milter-greylist README file: === If your resolver is not thread safe, install BIND9, and use
                      Message 10 of 15 , Sep 4, 2007
                      • 0 Attachment
                        Ronald F. Guilmette wrote:
                        > In message <20070903193047.4AF311F3EA1@...>,
                        > wietse@... (Wietse Venema) wrote:
                        >
                        >> Ronald F. Guilmette:
                        >>> Just for the sake of my edification, could you please also tell me
                        >>> _why_ a Postfix milter has to be thread safe?
                        >> There is no such thing as a Postfix Milter.
                        >>
                        >> Your milter application runs on top of libmilter.
                        >
                        > Well, that sort-of begs the question... What if it _didn't_ run on
                        > top of libmilter? What if it skipped that and tried to interface
                        > to Postfix smtpd directly? Would my milter still have to be
                        > thread-safe under that scenario?
                        >
                        > (Please forgive me for making such an issue of this, but it really
                        > would be a major pain in the ass to make what I have in mind thread-
                        > safe. A significant part of the problem with doing that... but
                        > certainly not the only problem... is that the spam filter that I'd
                        > like to turn into a milter does DNS lookups. Sadly, I am not aware
                        > of any thread-safe DNS primitives library, so I kind-of think that
                        > I'd be stuck trying to write my own, from scratch. Am I just looking
                        > in the Wrong Places? Does anybody here happen to know of a reasonably
                        > priced thread-safe DNS primitives library?)
                        >


                        are you confusing thread-safe with asynchronous?

                        from milter-greylist README file:

                        ===
                        If your resolver is not thread safe, install BIND9, and use
                        --with-libbind. If you know your resolver is thread-safe but
                        configure tells otherwise (because you lack the res_ninit() function),
                        then use --with-thread-safe-resolver.

                        If you install BIND9, make sure it includes libbind.a, since this is
                        what milter-greylist needs. libbind.a is not created in BIND9 default
                        build setup, so you might not have it in a precompiled package. If you
                        cannot find a package that contains libbind.a, then you have to rebuild
                        BIND9 from sources, using the --enable-libbind
                        flag to BIND9's configure.
                        ===
                      • Jose-Marcio Martins da Cruz
                        ... It seems to me that you don t need a milter architecture to implement what you re trying to do. But either way, no matter how you you implement your
                        Message 11 of 15 , Sep 5, 2007
                        • 0 Attachment
                          Ronald F. Guilmette wrote:
                          > In message <46DC7A95.8040503@...>,
                          > Michael Tokarev <mjt@...> wrote:

                          >
                          > That _may_ be the solution to my problem, but just as a general matter,
                          > I would greatly prefer not to have to worry about thread safety at all
                          > if I can avoid it. Also, programming in an event driven style is likely
                          > to be a good deal harder than just making simple calls to res_query()...
                          > as my filter does now... and then just waiting, synchronously, for the
                          > result to come back.

                          It seems to me that you don't need a milter architecture to implement
                          what you're trying to do.

                          But either way, no matter how you you implement your filter, you'll
                          surely need to implement some sort of concurrent access management at
                          some place. Unless you configure your MTA to not handle more than one
                          message at a time.

                          Also, Solaris (>= 8), FreeBSD (>= 5.3) and Linux (recent) resolvers are
                          thread safe. Don't know about others.


                          --
                          ---------------------------------------------------------------
                          Jose Marcio MARTINS DA CRUZ
                          Ecole des Mines de Paris http://j-chkmail.ensmp.fr
                          60, bd Saint Michel http://www.ensmp.fr/~martins
                          75272 - PARIS CEDEX 06 mailto:Jose-Marcio.Martins@...
                        • Leandro Santi
                          ... Before diving into the details of rolling milter by your own, take a look at Vernon Schryver s *excellent* DCC. It does DNS BLs, milter (using the
                          Message 12 of 15 , Sep 5, 2007
                          • 0 Attachment
                            On 9/4/07, Ronald F. Guilmette <rfg@...> wrote:
                            > >Your app can implement/talk milter protocol internally. You don't
                            > >interface to Postfix smtpd directly, but you use sockets to talk
                            > >to Postfix. Speaking of the protocol - well, one (reference)
                            > >implementation of server-side milter (Postfix is on client side
                            > >in this case) is libmilter, which use threads - you can roll your
                            > >own...
                            >
                            > Thanks. That's exactly what I suspected... and I may indeed have to do
                            > that.

                            Before diving into the details of rolling milter
                            by your own, take a look at Vernon Schryver's
                            *excellent* DCC. It does DNS BLs, milter (using
                            the reference milter API, so it's multithreaded),
                            and much more.

                            http://rhyolite.com/anti-spam/dcc/

                            (Look for dccm.) It's primarily written in C. From
                            what I recall, it works out the DNS queries by
                            wrapping the standard system resolver API with
                            POSIX thread locks (this applies to dccm, but not
                            to other single threaded apps that are being
                            distributed in the same source tarball). This
                            workaround should be good enough, given an
                            appropriate system architecture.

                            It might be worth to take a look at it (including
                            its license, which contains exceptional terms for
                            a specific case).

                            Leandro
                          • Leandro Santi
                            ... One more detail, newest dccm seem to implement the notion of helper DNS resolver processes (i.e. not threads,) so that the multithreaded dccm can sustain
                            Message 13 of 15 , Sep 5, 2007
                            • 0 Attachment
                              On 9/5/07, Leandro Santi <lesanti@...> wrote:
                              > http://rhyolite.com/anti-spam/dcc/
                              >
                              > (Look for dccm.) It's primarily written in C. From
                              > what I recall, it works out the DNS queries by
                              > wrapping the standard system resolver API with
                              > POSIX thread locks (this applies to dccm, but not
                              > to other single threaded apps that are being
                              > distributed in the same source tarball). This
                              > workaround should be good enough, given an
                              > appropriate system architecture.

                              One more detail, newest dccm seem to implement
                              the notion of "helper" DNS resolver processes (i.e.
                              not threads,) so that the multithreaded dccm can
                              sustain better thoughput. Unfortunately, I haven't
                              been using the DCC for a couple of years, so I'm
                              not up to finer details; but it might be worth to take
                              a look at it.

                              Leandro
                            Your message has been successfully submitted and would be delivered to recipients shortly.