Loading ...
Sorry, an error occurred while loading the content.

Re: postfix as incoming relay to protect exchange server / recipient lookup

Expand Messages
  • DTNX/NGMX Postmaster
    ... I would suggest that in most cases, the RAV option is probably the best choice, unless performance is an issue because of hardware constraints, or mail
    Message 1 of 30 , Dec 1, 2010
    View Source
    • 0 Attachment
      On 01/12/2010, at 23:18, Stan Hoeppner wrote:

      > Martin Kellermann put forth on 12/1/2010 9:19 AM:
      >
      >> so, is it still (seven years later) "The right thing™ to do" ?
      >> will it work proper with exchange 2007/2010 ?
      >> since the usage of "script-generated map-files" will never show
      >> a real-time picture of the valid exchange-recipients to postfix,
      >> isn't it nicer to do "online LDAP requests" from postfix?
      >> maybe this is possible with a LDAP-SASL plugin...?
      >
      > If you have very few users, say 1-100, and your organization doesn't
      > have frequent personnel changes, I recommend using relay_recipient_maps
      > and manually editing the table when needed.
      >
      > If more than that, for many reasons, I recommend using recipient address
      > verification instead of LDAP lookups, assuming you have decent spam
      > filtering techniques on your Postfix gateway, which is a requirement in
      > today's world anyway.
      >
      > http://www.postfix.org/ADDRESS_VERIFICATION_README.html
      > http://www.postfix.org/ADDRESS_VERIFICATION_README.html#recipient
      >
      > The main reasons I recommend this over LDAP are:
      >
      > 1. These probes are typically faster than LDAP queries
      > 2. Recipient verification caches probe results reducing query load
      > and increasing performance. AFAIK LDAP results aren't cached.
      > 3. _VASTLY_ simpler configuration compared to LDAP
      > 4. Doesn't require LDAP support be compiled into your Postfix package
      > 5. You get a _realtime_ answer regarding SMTP mailbox availability.
      > An LDAP response may differ from an Exchange SMTP response due to
      > a number of reasons, such as AD synchronization, etc. This is
      > probably rare, but it can happen.

      I would suggest that in most cases, the RAV option is probably the best choice, unless performance is an issue because of hardware constraints, or mail volume?

      Compared to maintaining a recipient map it is pretty much automatic once set up, and very resilient when it comes to changes to your Exchange server and/or AD servers. Which you'll love when you are not the person maintaining the Exchange server, or someone in upper management decides that all accounts should have aliases for all the common typos they can think of.

      Compared to LDAP it can be easily tested using any telnet client, does not depend on having a valid account within the AD forest, and is configured using the transport map entry you need anyway to deliver mail. You don't need any additional firewall rules either, beyond the port you use for SMTP traffic.

      We use RAV on our MX servers to route mail to clients with Exchange. Simple, and it works like a charm.

      Cya,
      Jona
    • Stan Hoeppner
      ... So a remote LDAP query is cheaper than a local table lookup? Interesting. I would have assumed lookups to the local RAV cache file would be infinitely
      Message 2 of 30 , Dec 1, 2010
      View Source
      • 0 Attachment
        Victor Duchovni put forth on 12/1/2010 5:06 PM:
        > On Wed, Dec 01, 2010 at 04:50:20PM -0600, Stan Hoeppner wrote:

        >> Are LDAP queries still simpler and cheaper once all recipient addresses
        >> are cached in $data_directory/verify_cache?
        >
        > Yes, because the vast majority of "RCPT TO" commands are dictionary
        > attacks, if not all the time, at least at peak loads when it matters.
        > Sending an SMTP probe is much more expensive than making an LDAP query.

        So a remote LDAP query is cheaper than a local table lookup?
        Interesting. I would have assumed lookups to the local RAV cache file
        would be infinitely faster than a remote LDAP query. I would guess that
        for many/most organizations the RAV cache would be populated within a
        few days max, if not a few hours. After that point, all lookups are to
        a local table, which again, I'd assume would be much faster than an LDAP
        query. But you're saying the remote LDAP query is "cheaper" in this
        case, Viktor?

        >> Do you disagree with my other 4 points Viktor? You know this stuff far
        >> better than I, so if I'm wrong on the other points I'd like to be
        >> corrected, so as not to make the same recommendations in the future.
        >
        > My comment is about LDAP table lookups vs. RAV (Recipient Address
        > Validation). I don't recall what your other points were, if it is not
        > critical, we probably don't need to revisit them.

        I don't know about being "critical", but I think they are valid points
        supporting the use of RAV.

        > LDAP tables are supported and not discouraged, but high volume sites
        > may want to dedicate some LDAP replicas to MTA queries.

        I'm not discouraging anyone from using LDAP queries. I merely made the
        case that many times RAV is a better choice, and stated some reasons why.

        I know of one Canadian company with 40K+ users worldwide and a few
        million MX connections a day that uses strictly RAV on their two MX
        relays. Their reasons for doing so have much more to do with ease and
        consistency of management than performance though. Mainly that the
        dozes of departmental mail servers run a mix of different MTAs
        (Exchange, Groupwise, Notes, etc) and directory services (AD,
        eDirectory, etc), making it too difficult to try managing a single LDAP
        master directory for the MX servers to query. Thus, they use RAV, and
        it works extremely well for them, from both a management and performance
        perspective.

        --
        Stan
      • Victor Duchovni
        ... The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary attacks always yield cache misses. ... You are forgetting that dictionary attacks
        Message 3 of 30 , Dec 1, 2010
        View Source
        • 0 Attachment
          On Wed, Dec 01, 2010 at 11:43:30PM -0600, Stan Hoeppner wrote:

          > Victor Duchovni put forth on 12/1/2010 5:06 PM:
          > > On Wed, Dec 01, 2010 at 04:50:20PM -0600, Stan Hoeppner wrote:
          >
          > >> Are LDAP queries still simpler and cheaper once all recipient addresses
          > >> are cached in $data_directory/verify_cache?
          > >
          > > Yes, because the vast majority of "RCPT TO" commands are dictionary
          > > attacks, if not all the time, at least at peak loads when it matters.
          > > Sending an SMTP probe is much more expensive than making an LDAP query.
          >
          > So a remote LDAP query is cheaper than a local table lookup?

          The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary
          attacks always yield cache misses.

          > Interesting. I would have assumed lookups to the local RAV cache file
          > would be infinitely faster than a remote LDAP query. I would guess that
          > for many/most organizations the RAV cache would be populated within a
          > few days max, if not a few hours.

          You are forgetting that dictionary attacks are almost exclusively queries
          for non-existent users. Think clearly, and think outside the box about
          worst-case behaviour.

          > But you're saying the remote LDAP query is "cheaper" in this
          > case, Viktor?

          Because I am not thinking about normal loads that don't matter. One
          needs to survive hostile loads.

          > > LDAP tables are supported and not discouraged, but high volume sites
          > > may want to dedicate some LDAP replicas to MTA queries.
          >
          > I'm not discouraging anyone from using LDAP queries. I merely made the
          > case that many times RAV is a better choice, and stated some reasons why.

          The reasons are not valid under hostile conditions.

          --
          Viktor.
        • Stan Hoeppner
          ... Well, yes, assuming you re not dropping such hostile connections with Postscreen, smtpd_foo_restrictions, and/or fail2ban or similar, _before_ the
          Message 4 of 30 , Dec 1, 2010
          View Source
          • 0 Attachment
            Victor Duchovni put forth on 12/1/2010 11:51 PM:
            > On Wed, Dec 01, 2010 at 11:43:30PM -0600, Stan Hoeppner wrote:
            >
            >> Victor Duchovni put forth on 12/1/2010 5:06 PM:
            >>> On Wed, Dec 01, 2010 at 04:50:20PM -0600, Stan Hoeppner wrote:
            >>
            >>>> Are LDAP queries still simpler and cheaper once all recipient addresses
            >>>> are cached in $data_directory/verify_cache?
            >>>
            >>> Yes, because the vast majority of "RCPT TO" commands are dictionary
            >>> attacks, if not all the time, at least at peak loads when it matters.
            >>> Sending an SMTP probe is much more expensive than making an LDAP query.
            >>
            >> So a remote LDAP query is cheaper than a local table lookup?
            >
            > The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary
            > attacks always yield cache misses.
            >
            >> Interesting. I would have assumed lookups to the local RAV cache file
            >> would be infinitely faster than a remote LDAP query. I would guess that
            >> for many/most organizations the RAV cache would be populated within a
            >> few days max, if not a few hours.
            >
            > You are forgetting that dictionary attacks are almost exclusively queries
            > for non-existent users. Think clearly, and think outside the box about
            > worst-case behaviour.
            >
            >> But you're saying the remote LDAP query is "cheaper" in this
            >> case, Viktor?
            >
            > Because I am not thinking about normal loads that don't matter. One
            > needs to survive hostile loads.
            >
            >>> LDAP tables are supported and not discouraged, but high volume sites
            >>> may want to dedicate some LDAP replicas to MTA queries.
            >>
            >> I'm not discouraging anyone from using LDAP queries. I merely made the
            >> case that many times RAV is a better choice, and stated some reasons why.
            >
            > The reasons are not valid under hostile conditions.

            Well, yes, assuming you're not dropping such hostile connections with
            Postscreen, smtpd_foo_restrictions, and/or fail2ban or similar, _before_
            the transactions get to the recipient validation stage in volume. Are
            you assuming most, or a significant number, of such transactions live to
            the RAV stage?

            I just noticed Viktor that you are one of the LDAP code authors. Please
            note I am making no arguments _against_ the use of LDAP, but merely
            making some arguments _for_ the use of RAV.

            --
            Stan
          • Jose-Marcio Martins da Cruz
            ... Just to illustrate with numbers, what Viktor says, a daily activity on one of our servers (not a big one) : from all connections reaching a RCPT To step,
            Message 5 of 30 , Dec 2, 2010
            View Source
            • 0 Attachment
              Victor Duchovni wrote:
              > On Wed, Dec 01, 2010 at 11:43:30PM -0600, Stan Hoeppner wrote:

              > The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary
              > attacks always yield cache misses.

              > You are forgetting that dictionary attacks are almost exclusively queries
              > for non-existent users. Think clearly, and think outside the box about
              > worst-case behaviour.

              > Because I am not thinking about normal loads that don't matter. One
              > needs to survive hostile loads.
              >

              Just to illustrate with numbers, what Viktor says, a daily activity on one of our servers (not a big
              one) : from all connections reaching a "RCPT To" step, 18313 are to real users and 76623 to non
              existing users. Well, this is a daily normal activity, not even a hostile load. Clearly, most checks
              are cache misses.
            • Stan Hoeppner
              ... Whether most would be cache misses would depend on how you have your verify(8) database cache configured (it does cache negative results), and the nature
              Message 6 of 30 , Dec 2, 2010
              View Source
              • 0 Attachment
                Jose-Marcio Martins da Cruz put forth on 12/2/2010 2:40 AM:
                > Victor Duchovni wrote:
                >> On Wed, Dec 01, 2010 at 11:43:30PM -0600, Stan Hoeppner wrote:
                >
                >> The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary
                >> attacks always yield cache misses.
                >
                >> You are forgetting that dictionary attacks are almost exclusively queries
                >> for non-existent users. Think clearly, and think outside the box about
                >> worst-case behaviour.
                >
                >> Because I am not thinking about normal loads that don't matter. One
                >> needs to survive hostile loads.
                >>
                >
                > Just to illustrate with numbers, what Viktor says, a daily activity on
                > one of our servers (not a big one) : from all connections reaching a
                > "RCPT To" step, 18313 are to real users and 76623 to non existing users.
                > Well, this is a daily normal activity, not even a hostile load. Clearly,
                > most checks are cache misses.

                Whether 'most' would be cache misses would depend on how you have your
                verify(8) database cache configured (it does cache negative results),
                and the nature of those 76623 connections. Search those 76623 and see
                how many are duplicates. It's probably alot more than you'd think,
                because spammers have been spamming to scraped message IDs for years.

                I'll have to do some checking, but I'm pretty sure that at $mydomain
                about 80% of all spam addressed to non existent users is addressed to
                scraped message IDs or butchered variants of them. This happens when
                spammers pass/sell their lists amongst one another and don't take any
                care when exporting/copying the address data. In the case of message ID
                spam, the verify(8) cache would work well. Of note, I've never seen an
                actual 'dictionary' attack on any of my MX machines. Note I didn't say
                they don't occur, only that I've yet to see one. I have seen dictionary
                attacks against FTP and SSH ports, but not SMTP. Given how fruitless
                they are, I would suspect even dumb spammers probably avoid dictionary
                spamming today.

                --
                Stan
              • Jose-Marcio Martins da Cruz
                ... I m not using verify. I m just telling the number of RCPT commands (18xxx + 76xxx), and how many of them resulted in user unknown (76xxx) and user OK
                Message 7 of 30 , Dec 2, 2010
                View Source
                • 0 Attachment
                  Stan Hoeppner wrote:
                  > Jose-Marcio Martins da Cruz put forth on 12/2/2010 2:40 AM:
                  >> Victor Duchovni wrote:
                  >>> On Wed, Dec 01, 2010 at 11:43:30PM -0600, Stan Hoeppner wrote:
                  >>> The lookup is always a cache miss. Then an SMTP probe is sent. Dictionary
                  >>> attacks always yield cache misses.
                  >>> You are forgetting that dictionary attacks are almost exclusively queries
                  >>> for non-existent users. Think clearly, and think outside the box about
                  >>> worst-case behaviour.
                  >>> Because I am not thinking about normal loads that don't matter. One
                  >>> needs to survive hostile loads.
                  >>>
                  >> Just to illustrate with numbers, what Viktor says, a daily activity on
                  >> one of our servers (not a big one) : from all connections reaching a
                  >> "RCPT To" step, 18313 are to real users and 76623 to non existing users.
                  >> Well, this is a daily normal activity, not even a hostile load. Clearly,
                  >> most checks are cache misses.
                  >
                  > Whether 'most' would be cache misses would depend on how you have your
                  > verify(8) database cache configured (it does cache negative results),
                  > and the nature of those 76623 connections. Search those 76623 and see
                  > how many are duplicates. It's probably alot more than you'd think,
                  > because spammers have been spamming to scraped message IDs for years.

                  I'm not using verify. I'm just telling the number of RCPT commands (18xxx + 76xxx), and how many of
                  them resulted in user unknown (76xxx) and user OK (18xxx). Should add those relaying denied and
                  other rejections.

                  On the other hand, real numbers should be much higher than I said as any SMTP commands are rejected
                  if the client do more than some number of RCPT errors. This is done by a milter and the potentially
                  real numbers aren't counted in the above.

                  So, to continue, I took longer times : 10 days, individually and 10 days all together. What is
                  constant is that from the list of unknown users, there are something like 1/4 are different
                  recipients, each day. But when I have the whole 10 days together, if falls just from ~25% to ~15%.
                  That means that the set of unique unknown recipients changes each day.

                  But indeed, and I insist to say, this is a normal situation. A hostile situation should be when
                  someone intentionnaly floods you with 10 times more than the usual "unknown users" rate, using
                  different recipients. You shall protect against this, not against normal situations. Still, my
                  numbers correspond to a small server.

                  As a comparable "side effect", in greylisting filters, you can think about the database of of
                  triplets waiting to be validated as some sort of cache. Well, some of these filters don't do
                  recipient check before adding pending entries in the database. A simple way to attack these filters
                  is to fill up their databases with a lot (a million or more) of entries which will never be
                  validated. Growing this "cache" without limits may not be a problem for small idle servers, but
                  indeed is for huge servers.


                  > I'll have to do some checking, but I'm pretty sure that at $mydomain
                  > about 80% of all spam addressed to non existent users is addressed to
                  > scraped message IDs or butchered variants of them. This happens when

                  Please, check. Speculate is good to begin thinking about something, but better is to verify if the
                  first idea is good or not.

                  > spammers pass/sell their lists amongst one another and don't take any
                  > care when exporting/copying the address data. In the case of message ID
                  > spam, the verify(8) cache would work well. Of note, I've never seen an
                  > actual 'dictionary' attack on any of my MX machines. Note I didn't say
                  > they don't occur, only that I've yet to see one. I have seen dictionary
                  > attacks against FTP and SSH ports, but not SMTP. Given how fruitless
                  > they are, I would suspect even dumb spammers probably avoid dictionary
                  > spamming today.

                  I see every day. I can't tell that one kind of pattern is more frequent than others (cluttered
                  variants of existing addresses, common first names, random user parts, ...). I guess it's almost the
                  same. I see all of them.
                • Martin Kellermann
                  ... thank you all for your detailed suggestions. looking at the pro-and-con s of LDAP vs RAV, i think, i will give RAV a try. since i never used the postfix
                  Message 8 of 30 , Dec 2, 2010
                  View Source
                  • 0 Attachment
                    On 02/12/2010, at 06:25, DTNX/NGMX Postmaster wrote:
                    > On 01/12/2010, at 23:18, Stan Hoeppner wrote:
                    >
                    >> Martin Kellermann put forth on 12/1/2010 9:19 AM:
                    >>
                    >>> so, is it still (seven years later) "The right thing™ to do" ?
                    >>> will it work proper with exchange 2007/2010 ?
                    >>> since the usage of "script-generated map-files" will never show
                    >>> a real-time picture of the valid exchange-recipients to postfix,
                    >>> isn't it nicer to do "online LDAP requests" from postfix?
                    >>> maybe this is possible with a LDAP-SASL plugin...?
                    >> If you have very few users, say 1-100, and your organization doesn't
                    >> have frequent personnel changes, I recommend using relay_recipient_maps
                    >> and manually editing the table when needed.
                    >>
                    >> If more than that, for many reasons, I recommend using recipient address
                    >> verification instead of LDAP lookups, assuming you have decent spam
                    >> filtering techniques on your Postfix gateway, which is a requirement in
                    >> today's world anyway.
                    >>
                    >> http://www.postfix.org/ADDRESS_VERIFICATION_README.html
                    >> http://www.postfix.org/ADDRESS_VERIFICATION_README.html#recipient
                    >>
                    >> The main reasons I recommend this over LDAP are:
                    >>
                    >> 1. These probes are typically faster than LDAP queries
                    >> 2. Recipient verification caches probe results reducing query load
                    >> and increasing performance. AFAIK LDAP results aren't cached.
                    >> 3. _VASTLY_ simpler configuration compared to LDAP
                    >> 4. Doesn't require LDAP support be compiled into your Postfix package
                    >> 5. You get a _realtime_ answer regarding SMTP mailbox availability.
                    >> An LDAP response may differ from an Exchange SMTP response due to
                    >> a number of reasons, such as AD synchronization, etc. This is
                    >> probably rare, but it can happen.
                    > I would suggest that in most cases, the RAV option is probably the best choice, unless performance is an issue because of hardware constraints, or mail volume?
                    >
                    > Compared to maintaining a recipient map it is pretty much automatic once set up, and very resilient when it comes to changes to your Exchange server and/or AD servers. Which you'll love when you are not the person maintaining the Exchange server, or someone in upper management decides that all accounts should have aliases for all the common typos they can think of.
                    >
                    > Compared to LDAP it can be easily tested using any telnet client, does not depend on having a valid account within the AD forest, and is configured using the transport map entry you need anyway to deliver mail. You don't need any additional firewall rules either, beyond the port you use for SMTP traffic.
                    >
                    > We use RAV on our MX servers to route mail to clients with Exchange. Simple, and it works like a charm.
                    >
                    > Cya,
                    > Jona
                    thank you all for your detailed suggestions.
                    looking at the pro-and-con's of LDAP vs RAV, i think, i will give RAV a try.

                    since i never used the postfix verify server before, the master.cf shows
                    the default
                    "verify unix - - n - 1 verify"

                    main.cf is unconfigured regarding the verify server, so it will use this
                    (postconf -d)
                    defaults:
                    address_verify_cache_cleanup_interval = 12h
                    address_verify_default_transport = $default_transport
                    address_verify_local_transport = $local_transport
                    address_verify_map = btree:$data_directory/verify_cache
                    address_verify_negative_cache = yes
                    address_verify_negative_expire_time = 3d
                    address_verify_negative_refresh_time = 3h
                    address_verify_poll_count = ${stress?1}${stress:3}
                    address_verify_poll_delay = 3s
                    address_verify_positive_expire_time = 31d
                    address_verify_positive_refresh_time = 7d
                    address_verify_relay_transport = $relay_transport
                    address_verify_relayhost = $relayhost
                    address_verify_sender = $double_bounce_sender
                    address_verify_sender_dependent_default_transport_maps =
                    $sender_dependent_default_transport_maps
                    address_verify_sender_dependent_relayhost_maps =
                    $sender_dependent_relayhost_maps
                    address_verify_service_name = verify
                    address_verify_transport_maps = $transport_maps
                    address_verify_virtual_transport = $virtual_transport

                    any hints or pitfalls using this values?

                    i tried testing RAV on one transport-domain by adding
                    "check_recipient_access hash:/etc/verifylist" to
                    smtpd_recipient_restrictions. in /etc/verifylist i defined "test.com
                    reject_unverified_recipient" and
                    it seems to work so far.
                    but i see a strange "double-bounce" in mail.log which i don't understand:
                    --------------
                    12:45:00 postfix/smtpd[26517]: connect from [...]
                    12:45:00 postfix/cleanup[26524]: 500E0EB438E:
                    message-id=<20101202114500.500E0EB438E@$myhostname>
                    12:45:01 postfix/qmgr[26504]: 500E0EB438E:
                    from=<double-bounce@$myhostname>, size=311, nrcpt=1 (queue active)
                    12:45:06 postfix/smtp[26774]: 500E0EB438E: to=<nx@...>,
                    relay=IP[IP]:PORT, delay=5.7, delays=0.6/0/0.03/5.1, dsn=5.1.1,
                    status=undeliverable (host IP[IP] said: 550 5.1.1 User unknown (in reply
                    to RCPT TO command))
                    12:45:06 postfix/smtpd[26517]: NOQUEUE: reject: RCPT from [...]: 450
                    4.1.1 <nx@...>: Recipient address rejected: unverified address:
                    host IP[IP] said: 550 5.1.1 User unknown (in reply to RCPT TO command);
                    from=<sender@...> to=<nx@...> proto=ESMTP helo=<[...]>
                    12:45:07 postfix/smtpd[26517]: disconnect from [...]
                    --------------
                    and there's a 5 sec. delay ... seems way too long to me for just
                    checking the recipient...!?


                    thank you

                    PS: should unverified_recipient_reject_code set to 450 or 550 ?
                  • Eero Volotinen
                    ... double-bounce is account used for validation of user account. -- Eero
                    Message 9 of 30 , Dec 2, 2010
                    View Source
                    • 0 Attachment
                      > but i see a strange "double-bounce" in mail.log which i don't understand:

                      double-bounce is account used for validation of user account.

                      --
                      Eero
                    • Martin Kellermann
                      ... thank you for explaining this... so everything seems to be fine so far... is this user name configurable?
                      Message 10 of 30 , Dec 2, 2010
                      View Source
                      • 0 Attachment
                        Am 02.12.2010 13:11, schrieb Eero Volotinen:
                        >> but i see a strange "double-bounce" in mail.log which i don't understand:
                        > double-bounce is account used for validation of user account.

                        thank you for explaining this... so everything seems to be fine so far...
                        is this user name configurable?
                      • Wietse Venema
                        ... Stan, if your server is connected to the internet, then your worst case will become your common case. Therefore it is a mistake to optimize the common
                        Message 11 of 30 , Dec 2, 2010
                        View Source
                        • 0 Attachment
                          Victor Duchovni:
                          > Because I am not thinking about normal loads that don't matter. One
                          > needs to survive hostile loads.
                          >
                          > > > LDAP tables are supported and not discouraged, but high volume sites
                          > > > may want to dedicate some LDAP replicas to MTA queries.
                          > >
                          > > I'm not discouraging anyone from using LDAP queries. I merely made the
                          > > case that many times RAV is a better choice, and stated some reasons why.
                          >
                          > The reasons are not valid under hostile conditions.

                          Stan, if your server is connected to the internet, then your worst
                          case will become your common case.

                          Therefore it is a mistake to optimize the common case.

                          Wietse
                        • Wietse Venema
                          ... That should be: it is a mistake to optimize the common case only. One word was lost in word smithing. Wietse
                          Message 12 of 30 , Dec 2, 2010
                          View Source
                          • 0 Attachment
                            Wietse Venema:
                            > Victor Duchovni:
                            > > Because I am not thinking about normal loads that don't matter. One
                            > > needs to survive hostile loads.
                            > >
                            > > > > LDAP tables are supported and not discouraged, but high volume sites
                            > > > > may want to dedicate some LDAP replicas to MTA queries.
                            > > >
                            > > > I'm not discouraging anyone from using LDAP queries. I merely made the
                            > > > case that many times RAV is a better choice, and stated some reasons why.
                            > >
                            > > The reasons are not valid under hostile conditions.
                            >
                            > Stan, if your server is connected to the internet, then your worst
                            > case will become your common case.
                            >
                            > Therefore it is a mistake to optimize the common case.

                            That should be: it is a mistake to optimize the common case only.

                            One word was lost in word smithing.

                            Wietse
                          • Stan Hoeppner
                            ... Yes, as always. I ve simply been looking at this from the premise that our countermeasures which stop spam connections before the RCPT TO stage will also
                            Message 13 of 30 , Dec 2, 2010
                            View Source
                            • 0 Attachment
                              Wietse Venema put forth on 12/2/2010 7:35 AM:
                              > Victor Duchovni:
                              >> Because I am not thinking about normal loads that don't matter. One
                              >> needs to survive hostile loads.
                              >>
                              >>>> LDAP tables are supported and not discouraged, but high volume sites
                              >>>> may want to dedicate some LDAP replicas to MTA queries.
                              >>>
                              >>> I'm not discouraging anyone from using LDAP queries. I merely made the
                              >>> case that many times RAV is a better choice, and stated some reasons why.
                              >>
                              >> The reasons are not valid under hostile conditions.
                              >
                              > Stan, if your server is connected to the internet, then your worst
                              > case will become your common case.
                              >
                              > Therefore it is a mistake to optimize the common case.
                              >
                              > Wietse

                              Yes, as always. I've simply been looking at this from the premise that
                              our countermeasures which stop spam connections before the RCPT TO stage
                              will also stop dictionary attacks before the RCPT TO stage since such
                              attacks typically come from the same types of sources. Everyone has
                              slightly different antispam countermeasures, so maybe this would account
                              for some folks seeing far more connections reach the RCPT TO stage than
                              others. Those using SA as a post queue filter, for instance, would
                              likely see far more of these making it to the RCPT TO stage. Am I
                              missing something?

                              "smtpd_delay_reject = yes" doesn't cause a user lookup for each
                              connection does it? Doesn't this merely log the RCPT TO address without
                              looking it up? If the latter, again, I'd assume antispam measures would
                              stop most of the dictionary attack RCPT TO queries from reaching the
                              downstream server via RAV. If I'm wrong here, please educate me.

                              --
                              Stan
                            • Stan Hoeppner
                              ... That delay should be no longer than what a typical delivery to the Exchange server would be. Since no message is sent, it should be shorter by quite a
                              Message 14 of 30 , Dec 2, 2010
                              View Source
                              • 0 Attachment
                                Martin Kellermann put forth on 12/2/2010 6:08 AM:

                                > and there's a 5 sec. delay ... seems way too long to me for just
                                > checking the recipient...!?

                                That delay should be no longer than what a typical delivery to the
                                Exchange server would be. Since no message is sent, it should be
                                shorter by quite a bit. I would guess the delay is within the Exchange
                                server, not Postfix, so you may need to do some sleuthing on the Exch
                                server to see what it causing the delay.

                                > PS: should unverified_recipient_reject_code set to 450 or 550 ?

                                You should probably leave this at the defaults. As I understand it, the
                                default configuration will return a 5xx for "unknown user" and a 4xx if
                                the query fails, due to network, etc.

                                --
                                Stan
                              • Wietse Venema
                                ... Some people record the sender and recipient, for the case when (not if) the countermeasures have the unavoidable false positive. ... As documented,
                                Message 15 of 30 , Dec 2, 2010
                                View Source
                                • 0 Attachment
                                  Stan Hoeppner:
                                  > Yes, as always. I've simply been looking at this from the premise that
                                  > our countermeasures which stop spam connections before the RCPT TO stage
                                  > will also stop dictionary attacks before the RCPT TO stage since such
                                  > attacks typically come from the same types of sources. ...

                                  Some people record the sender and recipient, for the case when (not
                                  if) the countermeasures have the unavoidable false positive.

                                  > "smtpd_delay_reject = yes" doesn't cause a user lookup for each
                                  > connection does it? Doesn't this merely log the RCPT TO address without
                                  > looking it up?

                                  As documented, smtpd_delay_reject changes the timing. It does
                                  not promise anything else.

                                  Wietse
                                • Victor Duchovni
                                  ... The OP is really far better off querying the LDAP server: server_host = ... suitable LDAP server or servers ... bind_dn = ... a low-value non-human AD
                                  Message 16 of 30 , Dec 2, 2010
                                  View Source
                                  • 0 Attachment
                                    On Thu, Dec 02, 2010 at 04:08:09PM -0600, Stan Hoeppner wrote:

                                    > Martin Kellermann put forth on 12/2/2010 6:08 AM:
                                    >
                                    > > and there's a 5 sec. delay ... seems way too long to me for just
                                    > > checking the recipient...!?
                                    >
                                    > That delay should be no longer than what a typical delivery to the
                                    > Exchange server would be. Since no message is sent, it should be
                                    > shorter by quite a bit. I would guess the delay is within the Exchange
                                    > server, not Postfix, so you may need to do some sleuthing on the Exch
                                    > server to see what it causing the delay.
                                    >
                                    > > PS: should unverified_recipient_reject_code set to 450 or 550 ?
                                    >
                                    > You should probably leave this at the defaults. As I understand it, the
                                    > default configuration will return a 5xx for "unknown user" and a 4xx if
                                    > the query fails, due to network, etc.

                                    The OP is really far better off querying the LDAP server:

                                    server_host = ... suitable LDAP server or servers ...
                                    bind_dn = ... a low-value non-human AD account, just for LDAP lookups ...
                                    bind_pw = ... the corresponding password ...
                                    search_base =
                                    scope = sub
                                    query_filter = proxyAddresses=smtp:%s
                                    result_attribute = mail

                                    In an upcoming snapshot the Postfix LDAP driver will be able to do GSSAPI
                                    auth to AD for those brave enough to try their hand at provisioning
                                    cross-platform Kerberos credential caches ...

                                    --
                                    Viktor.
                                  • Stan Hoeppner
                                    ... Completion of support for time stamps from different stages of message delivery. The information is now logged as delays=a/b/c/d where a=time before
                                    Message 17 of 30 , Dec 2, 2010
                                    View Source
                                    • 0 Attachment
                                      Martin Kellermann put forth on 12/2/2010 6:08 AM:

                                      > relay=IP[IP]:PORT, delay=5.7, delays=0.6/0/0.03/5.1, dsn=5.1.1,

                                      > --------------
                                      > and there's a 5 sec. delay ... seems way too long to me for just
                                      > checking the recipient...!?

                                      Completion of support for time stamps from different stages
                                      of message delivery. The information is now logged as
                                      "delays=a/b/c/d" where a=time before active queue, b=time
                                      in active queue, c=connection setup time, d=actual message
                                      delivery.

                                      Note that the bulk of your delay was during the "message delivery"
                                      phase. AFAIK verify(8) actions (RCPT TO) are included in this phase.

                                      If that was the very first RAV probe you'd obviously have some startup
                                      time delay, as the database file would be created for the first time,
                                      along with making the actual probe to the Exch server. Monitor these
                                      delays for a few days after you have RAV in production, and if those "d"
                                      delays are still high let us know.

                                      Also, if you are intending to test RAV performance before going into
                                      production, you need to perform at least one RAV query every 100s or the
                                      verify daemon will exit due to $max_idle = 100s. Once the daemon exits,
                                      the next RAV will suffer startup time, giving you unrealistic delay
                                      results compared to being in production. The startup time should
                                      typically be less than 1 second, but all the little delays add up.
                                      Eliminating all of these on the Postfix side will help narrow down
                                      delays being introduced on the Exch side.

                                      --
                                      Stan
                                    • Stan Hoeppner
                                      ... That may be Viktor. I think he should test both and pick the solution that works best in his environment, both from a performance and management
                                      Message 18 of 30 , Dec 2, 2010
                                      View Source
                                      • 0 Attachment
                                        Victor Duchovni put forth on 12/2/2010 4:27 PM:

                                        > The OP is really far better off querying the LDAP server:

                                        That may be Viktor. I think he should test both and pick the solution
                                        that works best in his environment, both from a performance and
                                        management perspective. Choice is usually a good thing, and he has
                                        plenty with Postfix. :)

                                        --
                                        Stan
                                      • mouss
                                        ... let s look at this from the exchange server viewpoint: - with ldap, exchange sees no (RAV) connections. - with RAV, exchange is hit for every address to
                                        Message 19 of 30 , Dec 5, 2010
                                        View Source
                                        • 0 Attachment
                                          Le 03/12/2010 01:55, Stan Hoeppner a écrit :
                                          > Victor Duchovni put forth on 12/2/2010 4:27 PM:
                                          >
                                          >> The OP is really far better off querying the LDAP server:
                                          >
                                          > That may be Viktor. I think he should test both and pick the solution
                                          > that works best in his environment, both from a performance and
                                          > management perspective. Choice is usually a good thing, and he has
                                          > plenty with Postfix. :)
                                          >

                                          let's look at this from the exchange server viewpoint:

                                          - with ldap, exchange sees no (RAV) connections.
                                          - with RAV, exchange is hit for every address to verify


                                          Given all the job that exchange does (or is supposed to do), and the
                                          costs of the licences if you need to add new servers, then you'd better
                                          hit the AD server.


                                          if you really want caching, then setup an intermediary postfix that does
                                          ldap lookup and hit it with RAV...
                                        • DTNX/NGMX Postmaster
                                          ... A testing tool like swaks is great for testing RAV and other SMTP transactions; http://jetmore.org/john/code/swaks/ ... The default is 450 for both
                                          Message 20 of 30 , Dec 5, 2010
                                          View Source
                                          • 0 Attachment
                                            On 02/12/2010, at 23:08, Stan Hoeppner wrote:

                                            > Martin Kellermann put forth on 12/2/2010 6:08 AM:
                                            >
                                            >> and there's a 5 sec. delay ... seems way too long to me for just
                                            >> checking the recipient...!?
                                            >
                                            > That delay should be no longer than what a typical delivery to the
                                            > Exchange server would be. Since no message is sent, it should be
                                            > shorter by quite a bit. I would guess the delay is within the Exchange
                                            > server, not Postfix, so you may need to do some sleuthing on the Exch
                                            > server to see what it causing the delay.

                                            A testing tool like 'swaks' is great for testing RAV and other SMTP transactions;

                                            http://jetmore.org/john/code/swaks/


                                            >> PS: should unverified_recipient_reject_code set to 450 or 550 ?
                                            >
                                            > You should probably leave this at the defaults. As I understand it, the
                                            > default configuration will return a 5xx for "unknown user" and a 4xx if
                                            > the query fails, due to network, etc.

                                            The default is '450' for both unknown users and transient errors, see;

                                            http://www.postfix.org/postconf.5.html#unverified_recipient_reject_code

                                            From the same page: "The unverified_recipient_reject_code parameter specifies the numerical response code when an address is known to bounce (default: 450, change into 550 when you are confident that it is safe to do so)".

                                            We also found it very handy to set up our 'notify_classes' parameter, so the postmaster (or any other user you configure) gets a transcript of the SMTP session when Postfix rejects mail. Note however that this adds load and can generate a large volume of messages to the postmaster. It works for us at our current volume, YMMV.

                                            Lastly, you may want to set a value for 'unverified_recipient_reject_reason' if you don't want to share the details of the backend server, such as hostname and/or IP address, with the outside world.

                                            Cya,
                                            Jona
                                          • DTNX/NGMX Postmaster
                                            ... Yes, it s the address_verify_sender parameter, see; http://www.postfix.org/postconf.5.html#address_verify_sender
                                            Message 21 of 30 , Dec 5, 2010
                                            View Source
                                            • 0 Attachment
                                              On 02/12/2010, at 13:19, Martin Kellermann wrote:

                                              > Am 02.12.2010 13:11, schrieb Eero Volotinen:
                                              >>> but i see a strange "double-bounce" in mail.log which i don't understand:
                                              >> double-bounce is account used for validation of user account.
                                              >
                                              > thank you for explaining this... so everything seems to be fine so far...
                                              > is this user name configurable?

                                              Yes, it's the 'address_verify_sender' parameter, see;

                                              http://www.postfix.org/postconf.5.html#address_verify_sender
                                              http://www.postfix.org/postconf.5.html#double_bounce_sender

                                              Ours is set to 'senderverification@...', since we use it for SAV as well, and that's where it's visible to others. We've seen similar setups from others as well, as well as the null sender address, 'postmaster' and so on. For use with your own backend (as RAV) it probably won't matter as much, but could still be useful for instant recognition while reading logs.

                                              HTH,
                                              Jona
                                            • DTNX/NGMX Postmaster
                                              ... This sounds a bit like premature optimization, which some say is the root of all evil. It also violates the Keep It Simple, Sysadmin principle ;-)
                                              Message 22 of 30 , Dec 5, 2010
                                              View Source
                                              • 0 Attachment
                                                On 05/12/2010, at 18:19, mouss wrote:

                                                > Le 03/12/2010 01:55, Stan Hoeppner a écrit :
                                                >> Victor Duchovni put forth on 12/2/2010 4:27 PM:
                                                >>
                                                >>> The OP is really far better off querying the LDAP server:
                                                >>
                                                >> That may be Viktor. I think he should test both and pick the solution
                                                >> that works best in his environment, both from a performance and
                                                >> management perspective. Choice is usually a good thing, and he has
                                                >> plenty with Postfix. :)
                                                >
                                                > let's look at this from the exchange server viewpoint:
                                                >
                                                > - with ldap, exchange sees no (RAV) connections.
                                                > - with RAV, exchange is hit for every address to verify
                                                >
                                                > Given all the job that exchange does (or is supposed to do), and the costs of the licences if you need to add new servers, then you'd better hit the AD server.
                                                >
                                                > if you really want caching, then setup an intermediary postfix that does ldap lookup and hit it with RAV...

                                                This sounds a bit like premature optimization, which some say is the root of all evil. It also violates the 'Keep It Simple, Sysadmin' principle ;-) Exchange isn't the most efficient mail server, but I'd suggest that, for the majority of Exchange deployments, you probably need to look elsewhere if the simple SMTP transactions iniated by RAV are causing a performance problem.

                                                In our case, most of the unwanted connections never make it to the RAV stage, as it's one of the last checks done, and the majority of all remaining connections seems to hit the local cache. As far as I'm aware we see very few SMTP dictionary attacks, and they all tend to bounce off one of the earlier verification steps. A 'check_recipient_access' map with known exceptions for example, such as deactivated accounts, the usual suspects such as 'iamjustsendingthisleter' and so on.

                                                Of course, YMMV. I agree with Stan, test it and keep what works best for your setup.

                                                Cya,
                                                Jona
                                              • mouss
                                                ... so, using RAV is more kiss than using ldap? let s see: - with ldap: postfix - AD - with rav: postfix - exchange - AD with RAV, you re adding a piece,
                                                Message 23 of 30 , Dec 5, 2010
                                                View Source
                                                • 0 Attachment
                                                  Le 05/12/2010 21:45, DTNX/NGMX Postmaster a écrit :
                                                  > On 05/12/2010, at 18:19, mouss wrote:
                                                  >
                                                  >> Le 03/12/2010 01:55, Stan Hoeppner a écrit :
                                                  >>> Victor Duchovni put forth on 12/2/2010 4:27 PM:
                                                  >>>
                                                  >>>> The OP is really far better off querying the LDAP server:
                                                  >>>
                                                  >>> That may be Viktor. I think he should test both and pick the solution
                                                  >>> that works best in his environment, both from a performance and
                                                  >>> management perspective. Choice is usually a good thing, and he has
                                                  >>> plenty with Postfix. :)
                                                  >>
                                                  >> let's look at this from the exchange server viewpoint:
                                                  >>
                                                  >> - with ldap, exchange sees no (RAV) connections.
                                                  >> - with RAV, exchange is hit for every address to verify
                                                  >>
                                                  >> Given all the job that exchange does (or is supposed to do), and the costs of the licences if you need to add new servers, then you'd better hit the AD server.
                                                  >>
                                                  >> if you really want caching, then setup an intermediary postfix that does ldap lookup and hit it with RAV...
                                                  >
                                                  > This sounds a bit like premature optimization, which some say is the root of all evil. It also violates the 'Keep It Simple, Sysadmin' principle ;-) Exchange isn't the most efficient mail server, but I'd suggest that, for the majority of Exchange deployments, you probably need to look elsewhere if the simple SMTP transactions iniated by RAV are causing a performance problem.
                                                  >

                                                  so, using RAV is more "kiss" than using ldap? let's see:

                                                  - with ldap: postfix -> AD
                                                  - with rav: postfix -> exchange -> AD

                                                  with RAV, you're adding a piece, and not a simple one.


                                                  anyway, I understand that different people have diferent opinions. so
                                                  let's move on...

                                                  > In our case, most of the unwanted connections never make it to the RAV stage, as it's one of the last checks done, and the majority of all remaining connections seems to hit the local cache. As far as I'm aware we see very few SMTP dictionary attacks, and they all tend to bounce off one of the earlier verification steps. A 'check_recipient_access' map with known exceptions for example, such as deactivated accounts, the usual suspects such as 'iamjustsendingthisleter' and so on.
                                                  >
                                                  > Of course, YMMV. I agree with Stan, test it and keep what works best for your setup.
                                                  >
                                                  > Cya,
                                                  > Jona
                                                • Martin Kellermann
                                                  ... yes, it was on the exchange side - the delay is configurable per domain. thanks.
                                                  Message 24 of 30 , Dec 6, 2010
                                                  View Source
                                                  • 0 Attachment
                                                    Am 02.12.2010 23:08, schrieb Stan Hoeppner:
                                                    > Martin Kellermann put forth on 12/2/2010 6:08 AM:
                                                    >
                                                    >> and there's a 5 sec. delay ... seems way too long to me for just
                                                    >> checking the recipient...!?
                                                    > That delay should be no longer than what a typical delivery to the
                                                    > Exchange server would be. Since no message is sent, it should be
                                                    > shorter by quite a bit. I would guess the delay is within the Exchange
                                                    > server, not Postfix, so you may need to do some sleuthing on the Exch
                                                    > server to see what it causing the delay.
                                                    yes, it was on the exchange side - the delay is configurable per domain.
                                                    thanks.
                                                    >> PS: should unverified_recipient_reject_code set to 450 or 550 ?
                                                    > You should probably leave this at the defaults. As I understand it, the
                                                    > default configuration will return a 5xx for "unknown user" and a 4xx if
                                                    > the query fails, due to network, etc.
                                                    >
                                                  • Martin Kellermann
                                                    ... yes, thanks ... we did already. every parameter is well documented - and after i read the postfix-manual-pages it s meanings are easy to understand :-)
                                                    Message 25 of 30 , Dec 6, 2010
                                                    View Source
                                                    • 0 Attachment
                                                      Am 05.12.2010 20:40, schrieb DTNX/NGMX Postmaster:
                                                      > On 02/12/2010, at 23:08, Stan Hoeppner wrote:
                                                      >
                                                      >> Martin Kellermann put forth on 12/2/2010 6:08 AM:
                                                      >>
                                                      >>> and there's a 5 sec. delay ... seems way too long to me for just
                                                      >>> checking the recipient...!?
                                                      >> That delay should be no longer than what a typical delivery to the
                                                      >> Exchange server would be. Since no message is sent, it should be
                                                      >> shorter by quite a bit. I would guess the delay is within the Exchange
                                                      >> server, not Postfix, so you may need to do some sleuthing on the Exch
                                                      >> server to see what it causing the delay.
                                                      > A testing tool like 'swaks' is great for testing RAV and other SMTP transactions;
                                                      >
                                                      > http://jetmore.org/john/code/swaks/
                                                      >
                                                      >
                                                      >>> PS: should unverified_recipient_reject_code set to 450 or 550 ?
                                                      >> You should probably leave this at the defaults. As I understand it, the
                                                      >> default configuration will return a 5xx for "unknown user" and a 4xx if
                                                      >> the query fails, due to network, etc.
                                                      > The default is '450' for both unknown users and transient errors, see;
                                                      >
                                                      > http://www.postfix.org/postconf.5.html#unverified_recipient_reject_code
                                                      >
                                                      > From the same page: "The unverified_recipient_reject_code parameter specifies the numerical response code when an address is known to bounce (default: 450, change into 550 when you are confident that it is safe to do so)".
                                                      >
                                                      > We also found it very handy to set up our 'notify_classes' parameter, so the postmaster (or any other user you configure) gets a transcript of the SMTP session when Postfix rejects mail. Note however that this adds load and can generate a large volume of messages to the postmaster. It works for us at our current volume, YMMV.
                                                      >
                                                      > Lastly, you may want to set a value for 'unverified_recipient_reject_reason' if you don't want to share the details of the backend server, such as hostname and/or IP address, with the outside world.
                                                      >
                                                      yes, thanks ... we did already.
                                                      every parameter is well documented - and after i read the
                                                      postfix-manual-pages it's meanings
                                                      are easy to understand :-)

                                                      setup works now as expected and we started a test-run to see how
                                                      performance on exchange
                                                      side and how big the SMTP-overhead is...

                                                      thanks again for all your effords and contributions.
                                                      > Cya,
                                                      > Jona
                                                    Your message has been successfully submitted and would be delivered to recipients shortly.