Loading ...
Sorry, an error occurred while loading the content.

Postfix counters (was: limit and monitor too many sasl login from same user)

Expand Messages
  • Wietse Venema
    ... I took a quick look into implementing this, but unfortunately, it is not as simple as I thought it would be. Thus, for a quick fix I suggest that you stick
    Message 1 of 13 , Oct 5, 2013
    • 0 Attachment
      Wietse Venema:
      > nik600:
      > > Virus, botnet and user's bad policies about password allows many 3rd party
      > > entities to stole passwords, in the last month i've experienced a grows of
      > > hacked users, and in some case many spam messages are sent from my servers
      > > before i can block the user.
      > >
      > > I've tried many combination
      > >
      > > smtpd_client_message_rate_limit
      > > smtpd_client_recipient_rate_limit
      > > anvil_rate_time_unit
      > >
      > > config options but as the sender changes frequently the client, sending
      > > from different locations, so the limits above has no effect and i can't
      > > stop the spammer.
      > >
      > > Does exists any configuration to limit the # of sasl login from an user?
      > >
      > > It could be very useful, and cloud be also useful monitor many login of the
      > > same sasl user from different ip.
      > >
      > > What do you thing about that?
      >
      > Who has time? Technically it's the same problem as client, message
      > and recipient rate limits.

      I took a quick look into implementing this, but unfortunately, it
      is not as simple as I thought it would be.

      Thus, for a quick fix I suggest that you stick with policyd,
      postfwd and similar solutions.

      What follows are some ideas to improve, generalize and simplify
      Postfix's built-in rate limit features.

      The existing smtpd_client_*whatever*_limit features are implemented
      with counters per SMTP client (specifically, the per-client connection,
      starttls, message and recipient rate limits).

      What you need is a rate limit per login name. Obviously, that
      requires a counter per login name.

      It should be easy enough to count per "login name" instead of per
      "SMTP client" (after all, those labels are just simple strings that
      select a hash-table entry).

      However it should not be too easy to exhaust server memory.

      In particular, Postfix must not try to maintain huge numbers of
      counters when some spammer tries a huge number of different login
      names in a short time. Either the use of per "login name" counters
      should be restricted to "known" logins, or there should be some
      caching mechanism that drops entries when the cache becomes full.

      Likewise, rate limits per sender email address or recipient email
      address sound attractive, but they should not use up insane amounts
      of memory for counters. Again, these counters should be restricted
      to "known" addresses, or there should be some caching mechanism
      that drops entries when the cache becomes full.

      These "new" rate limits would be more usable when they are supported
      by a cache that drops entries when full. Supporting this calls for
      new Postfix infrastructure: a counter table. I think that memcache
      would be a perfect solution for this job: it is memory-resident and
      therefore fast, it is designed to work within a finite budget, and
      it can be shared between different servers. One just needs to
      manage firewalling rules carefully.

      A "counter table" implemented on top of memcache would also provide
      a way to get rid of the anvil daemon.

      Wietse
    • Viktor Dukhovni
      ... Which requires a large number of concurrently compromised accounts. In most cases a spammer will have compromised a modest number of accounts. Since
      Message 2 of 13 , Oct 5, 2013
      • 0 Attachment
        On Sat, Oct 05, 2013 at 09:59:23AM -0400, Wietse Venema wrote:

        > It should be easy enough to count per "login name" instead of per
        > "SMTP client" (after all, those labels are just simple strings that
        > select a hash-table entry).
        >
        > However it should not be too easy to exhaust server memory.
        >
        > In particular, Postfix must not try to maintain huge numbers of
        > counters when some spammer tries a huge number of different login
        > names in a short time.

        Which requires a large number of concurrently compromised accounts.
        In most cases a spammer will have compromised a modest number of
        accounts. Since per-login message rates are not measured before
        successful authentication, I don't think this is a significant
        issue.

        > Either the use of per "login name" counters
        > should be restricted to "known" logins,

        This is for free, there is no such thing as an "unknown login".

        --
        Viktor.
      • Wietse Venema
        ... No. Think brute force account guessing attack . For example, a spammer tries (a long list of usernames) x (a long list of passwords) distributed over
        Message 3 of 13 , Oct 5, 2013
        • 0 Attachment
          Viktor Dukhovni:
          > On Sat, Oct 05, 2013 at 09:59:23AM -0400, Wietse Venema wrote:
          >
          > > It should be easy enough to count per "login name" instead of per
          > > "SMTP client" (after all, those labels are just simple strings that
          > > select a hash-table entry).
          > >
          > > However it should not be too easy to exhaust server memory.
          > >
          > > In particular, Postfix must not try to maintain huge numbers of
          > > counters when some spammer tries a huge number of different login
          > > names in a short time.
          >
          > Which requires a large number of concurrently compromised accounts.
          > In most cases a spammer will have compromised a modest number of

          No. Think "brute force account guessing attack". For example, a
          spammer tries (a long list of usernames) x (a long list of passwords)
          distributed over multiple compromised clients.

          Regardless of whether this is a common mode of operation, Postfix
          must not run out of memory when it happens.

          > > Either the use of per "login name" counters
          > > should be restricted to "known" logins,
          >
          > This is for free, there is no such thing as an "unknown login".

          Not true when "per login name" counters are updated regardless of
          whether the login exists, for example as part of a defense against
          brute-force account guessing attacks such as described above.

          In the general case of counters per sender address, recipient
          address, or some other information that the client provides, Postfix
          does not necessarily have ground truth of what is "known" (for
          example Postfix has no knowledge of all "known" email addresses on
          the Internet). Yet it could be useful to throttle down traffic that
          is obviously out of kilter.

          Did you have more ideas about shared-memory counter in memcache?

          Wietse
        • Patrick Ben Koetter
          ... How would you detect such an attack? A pattern of connection/login failures? A regular client should try x attempts within y and then give up, shouldn t
          Message 4 of 13 , Oct 5, 2013
          • 0 Attachment
            * Wietse Venema <postfix-users@...>:
            > Viktor Dukhovni:
            > > On Sat, Oct 05, 2013 at 09:59:23AM -0400, Wietse Venema wrote:
            > >
            > > > It should be easy enough to count per "login name" instead of per
            > > > "SMTP client" (after all, those labels are just simple strings that
            > > > select a hash-table entry).
            > > >
            > > > However it should not be too easy to exhaust server memory.
            > > >
            > > > In particular, Postfix must not try to maintain huge numbers of
            > > > counters when some spammer tries a huge number of different login
            > > > names in a short time.
            > >
            > > Which requires a large number of concurrently compromised accounts.
            > > In most cases a spammer will have compromised a modest number of
            >
            > No. Think "brute force account guessing attack". For example, a
            > spammer tries (a long list of usernames) x (a long list of passwords)
            > distributed over multiple compromised clients.
            >
            > Regardless of whether this is a common mode of operation, Postfix
            > must not run out of memory when it happens.

            How would you detect such an attack? A pattern of connection/login failures? A
            regular client should try x attempts within y and then give up, shouldn't it?
            Or do they try until someone manually intervenes?

            Can we assume such a feature would only be used on ports that have MUA to MTA
            traffic? On such a port could we separate spammer clients from regular
            clients? Do regular clients have behaviours that make them distinguishable
            from irregular (spammers) ones?

            If a regular client ended after x attempts within y time, should any further
            attempt lead to a ban, because it identifies an irregular client that keeps on
            failing?

            Also: A deep inspection (time consuming) could lookup the submitted password in
            <https://leakdb.abusix.com/info.html> and use the fact that there are matches
            to come to a decision.

            p@rick


            --
            [*] sys4 AG

            http://sys4.de, +49 (89) 30 90 46 64
            Franziskanerstraße 15, 81669 München

            Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
            Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
            Aufsichtsratsvorsitzender: Florian Kirstein
          • Patrick Ben Koetter
            ... Geolocation + day + x failed logins = attacker? IIRC you showed during the inital postscreen development that you were able to correlate spambots with
            Message 5 of 13 , Oct 5, 2013
            • 0 Attachment
              * Patrick Ben Koetter <postfix-users@...>:
              > * Wietse Venema <postfix-users@...>:
              > > Viktor Dukhovni:
              > > > On Sat, Oct 05, 2013 at 09:59:23AM -0400, Wietse Venema wrote:
              > > >
              > > > > It should be easy enough to count per "login name" instead of per
              > > > > "SMTP client" (after all, those labels are just simple strings that
              > > > > select a hash-table entry).
              > > > >
              > > > > However it should not be too easy to exhaust server memory.
              > > > >
              > > > > In particular, Postfix must not try to maintain huge numbers of
              > > > > counters when some spammer tries a huge number of different login
              > > > > names in a short time.
              > > >
              > > > Which requires a large number of concurrently compromised accounts.
              > > > In most cases a spammer will have compromised a modest number of
              > >
              > > No. Think "brute force account guessing attack". For example, a
              > > spammer tries (a long list of usernames) x (a long list of passwords)
              > > distributed over multiple compromised clients.
              > >
              > > Regardless of whether this is a common mode of operation, Postfix
              > > must not run out of memory when it happens.
              >
              > How would you detect such an attack? A pattern of connection/login failures? A
              > regular client should try x attempts within y and then give up, shouldn't it?
              > Or do they try until someone manually intervenes?
              >
              > Can we assume such a feature would only be used on ports that have MUA to MTA
              > traffic? On such a port could we separate spammer clients from regular
              > clients? Do regular clients have behaviours that make them distinguishable
              > from irregular (spammers) ones?
              >
              > If a regular client ended after x attempts within y time, should any further
              > attempt lead to a ban, because it identifies an irregular client that keeps on
              > failing?
              >
              > Also: A deep inspection (time consuming) could lookup the submitted password in
              > <https://leakdb.abusix.com/info.html> and use the fact that there are matches
              > to come to a decision.

              Geolocation + day + x failed logins = attacker? IIRC you showed during the
              inital postscreen development that you were able to correlate spambots with
              geolocation. The reason there were so many connections from spambots was it
              was day in these geolocations and people were up and had turned their
              compromised machines on.

              One could also track behaviour and build an IP reputation for login/password
              combinations. Connections from unusal IPs + repeated login failures could be a
              good reason to ban the IP.

              You could go fancy and use REPUTE <https://datatracker.ietf.org/wg/repute/> to
              store/retrieve reputation information. Finding connections from a banned IP in
              a database shared between different services (SMTP/POP/IMAP) might be useful
              for all participating services.

              p@rick

              --
              [*] sys4 AG

              http://sys4.de, +49 (89) 30 90 46 64
              Franziskanerstraße 15, 81669 München

              Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
              Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
              Aufsichtsratsvorsitzender: Florian Kirstein
            • Wietse Venema
              ... My idea: if Postfix can count something, then applications will follow. Whether it is counting login names, sender addresses, recipient addresses or
              Message 6 of 13 , Oct 5, 2013
              • 0 Attachment
                Patrick Ben Koetter:
                > * Wietse Venema <postfix-users@...>:
                > > Viktor Dukhovni:
                > > > On Sat, Oct 05, 2013 at 09:59:23AM -0400, Wietse Venema wrote:
                > > >
                > > > > It should be easy enough to count per "login name" instead of per
                > > > > "SMTP client" (after all, those labels are just simple strings that
                > > > > select a hash-table entry).
                > > > >
                > > > > However it should not be too easy to exhaust server memory.
                > > > >
                > > > > In particular, Postfix must not try to maintain huge numbers of
                > > > > counters when some spammer tries a huge number of different login
                > > > > names in a short time.
                > > >
                > > > Which requires a large number of concurrently compromised accounts.
                > > > In most cases a spammer will have compromised a modest number of
                > >
                > > No. Think "brute force account guessing attack". For example, a
                > > spammer tries (a long list of usernames) x (a long list of passwords)
                > > distributed over multiple compromised clients.
                > >
                > > Regardless of whether this is a common mode of operation, Postfix
                > > must not run out of memory when it happens.
                >
                > How would you detect such an attack? A pattern of connection/login
                > failures? A regular client should try x attempts within y and then
                > give up, shouldn't it? Or do they try until someone manually
                > intervenes?

                My idea: if Postfix can count something, then applications will
                follow. Whether it is counting login names, sender addresses,
                recipient addresses or something that I haven't thought of yet.

                > Can we assume such a feature would only be used on ports that have
                > MUA to MTA traffic? On such a port could we separate spammer clients
                > from regular clients? Do regular clients have behaviours that make
                > them distinguishable from irregular (spammers) ones?

                I suppose SMTP-related counters would be driven by smtpd(8), and
                Postfix still uses the same program for both MUA and MTA traffic.

                > If a regular client ended after x attempts within y time, should
                > any further attempt lead to a ban, because it identifies an irregular
                > client that keeps on failing?

                I hope that simple counters and limits can deal with this. If not,
                then making counters available via memcached would allow other
                programs to make more complex inferences.

                Unfortunately, digging up the login name (without waiting for a
                successful login) requires that Postfix implements parts of the
                SASL protocol itself. So that is not a very good application of
                counters for failed login attempts on known names, or login attempts
                on non-existent names.

                > Also: A deep inspection (time consuming) could lookup the submitted
                > password in <https://leakdb.abusix.com/info.html> and use the fact
                > that there are matches to come to a decision.

                Grabbing SASL password information with smtpd(8) would require more
                duplication of SASL inside smtpd(8).

                Wietse
              • Viktor Dukhovni
                ... With SASL we don t know what the login name is unless authentication succeeds. The user name encoding is mechanism specific (with GSSAPI it is in the
                Message 7 of 13 , Oct 5, 2013
                • 0 Attachment
                  On Sat, Oct 05, 2013 at 05:55:49PM -0400, Wietse Venema wrote:

                  > > > Either the use of per "login name" counters
                  > > > should be restricted to "known" logins,
                  > >
                  > > This is for free, there is no such thing as an "unknown login".
                  >
                  > Not true when "per login name" counters are updated regardless of
                  > whether the login exists, for example as part of a defense against
                  > brute-force account guessing attacks such as described above.

                  With SASL we don't know what the login name is unless authentication
                  succeeds. The user name encoding is mechanism specific (with GSSAPI
                  it is in the ticket!). Does the SASL API expose a user name for
                  failed logins?

                  > Did you have more ideas about shared-memory counter in memcache?

                  I have not given any thought to anything in this space that is not
                  SASL. My conjecture is that SASL is special.

                  The basic control for submission clients (be it by client address,
                  or by successful client login) is to substantially limit the
                  connection concurrency and rate from any given whitelisted IP or
                  any authorized account, and then to set a maximum submission message
                  rate per single connection. After that promptly close down any
                  access that is abused.

                  Don't use POP before SMTP, it is way past its prime.

                  --
                  Viktor.
                • Wietse Venema
                  ... See discussion in my reply to Patrick. ... I see this as a general counting mechanism with a custom policy on top. In this case, the number of logins on
                  Message 8 of 13 , Oct 6, 2013
                  • 0 Attachment
                    Viktor Dukhovni:
                    > With SASL we don't know what the login name is unless authentication
                    > succeeds. The user name encoding is mechanism specific (with GSSAPI
                    > it is in the ticket!). Does the SASL API expose a user name for
                    > failed logins?

                    See discussion in my reply to Patrick.

                    > > Did you have more ideas about shared-memory counter in memcache?
                    >
                    > I have not given any thought to anything in this space that is not
                    > SASL. My conjecture is that SASL is special.

                    I see this as a general counting mechanism with a custom policy on
                    top. In this case, the number of logins on the same account is
                    maintained by the counter, and the decision to block mail or disable
                    an account is policy. A general counting mechanism can be used for
                    other things: senders, recipients, etc. Many of my specific examples
                    are already implemented by policy daemons.

                    > The basic control for submission clients (be it by client address,
                    > or by successful client login) is to substantially limit the
                    > connection concurrency and rate from any given whitelisted IP or
                    > any authorized account, and then to set a maximum submission message
                    > rate per single connection. After that promptly close down any
                    > access that is abused.
                    >
                    > Don't use POP before SMTP, it is way past its prime.

                    Sure. We're talking about different parts of the solution: counters
                    (me) and policy enforcement (you). My immediate focus was on counters
                    as a way to enable new functionality such as Postfix health monitoring,
                    and making the anvil daemon redundant.

                    Wietse
                  Your message has been successfully submitted and would be delivered to recipients shortly.