Loading ...
Sorry, an error occurred while loading the content.

Make smtpd/Postscreen compatible with load balancers

Expand Messages
  • David Touzeau
    Dear We are facing an problem that we cannot resolve... Our main goal is to implement a load-balancer in front of Postfix (HaProxy). We have made a discuss
    Message 1 of 17 , May 27, 2012
    • 0 Attachment
      Dear

      We are facing an problem that we cannot resolve...
      Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
      We have made a discuss with HaProxy founder in order to implement the
      XCLIENT protcol but this is difficult for him to implement such protocol.

      In other way it seems that PostScreen is not really compatible with the
      proxy protocol.

      src/postscreen/postscreen.c :

      /* BUGS
      /* The \fBpostscreen\fR(8) built-in SMTP protocol engine
      /* currently does not announce support for AUTH, XCLIENT or
      /* XFORWARD.
      /* Support for AUTH may be added in the future.


      In fact we need to Postfix be compatible with Proxy protocol in order to store the IP address at each ends of the socket in the VSTREAM descriptor.
      With this way, an other handler, before PostScreen should get the IP/Port.

      We asking if there is a way to implement a load-balancer such as HaProxy or Crossroads without loosing IP addresses sources in order to make PostScreen available or any blacklisting feature..

      Best regards.
    • DTNX Postmaster
      ... I am a tad confused; why would your load balancer need to implement the XCLIENT protocol, when load balancing SMTP generally happens at one of the layers
      Message 2 of 17 , May 27, 2012
      • 0 Attachment
        On May 27, 2012, at 10:25, David Touzeau wrote:

        > We are facing an problem that we cannot resolve...
        > Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
        > We have made a discuss with HaProxy founder in order to implement the XCLIENT protcol but this is difficult for him to implement such protocol.
        >
        > In other way it seems that PostScreen is not really compatible with the proxy protocol.
        >
        > src/postscreen/postscreen.c :
        >
        > /* BUGS
        > /* The \fBpostscreen\fR(8) built-in SMTP protocol engine
        > /* currently does not announce support for AUTH, XCLIENT or
        > /* XFORWARD.
        > /* Support for AUTH may be added in the future.
        >
        >
        > In fact we need to Postfix be compatible with Proxy protocol in order to store the IP address at each ends of the socket in the VSTREAM descriptor.
        > With this way, an other handler, before PostScreen should get the IP/Port.
        >
        > We asking if there is a way to implement a load-balancer such as HaProxy or Crossroads without loosing IP addresses sources in order to make PostScreen available or any blacklisting feature..

        I am a tad confused; why would your load balancer need to implement the
        XCLIENT protocol, when load balancing SMTP generally happens at one of
        the layers below that? Typically at layer 3 or 4, if I am not mistaken?

        Also, why would you need a proxy? Unlike HTTP, SMTP does not benefit
        from front-end caching and the like. HTTP is a stateless, SMTP a
        stateful protocol? It is easily made highly available, just based on
        the MX records, and you can implement basic load balancing by having
        several MX records with the same priority, IIRC.

        In other words, what is the core problem you are trying to solve?

        Cya,
        Jona
      • Wietse Venema
        ... FYI nginx implements XCLIENT (and more). But it does not matter, since postscreen has no proxy support. ... First, there needs to be a configuration
        Message 3 of 17 , May 27, 2012
        • 0 Attachment
          David Touzeau:
          > Dear
          >
          > We are facing an problem that we cannot resolve...
          > Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
          > We have made a discuss with HaProxy founder in order to implement the
          > XCLIENT protcol but this is difficult for him to implement such protocol.

          FYI nginx implements XCLIENT (and more). But it does not matter,
          since postscreen has no proxy support.

          > In other way it seems that PostScreen is not really compatible with the
          > proxy protocol.

          First, there needs to be a configuration parameter that tells
          postscreen what kind of proxy command will be prepended to the
          client's SMTP stream.

          /etc/postfix/main.cf:
          # Is there a better name than "proxy protocol"?
          postscreen_proxy_protocol = whatever

          With this, postscreen will drop connections that fail to conform
          to the configured protocol.

          Regardless of command format details, if the proxy prepends a command
          to the client's SMTP stream, then postscreen must use unbuffered
          I/O to read that command. If buffering were turned on, the buffering
          layer could read past the proxy's <CR><LF> and eat up part of the
          client input kind-of like CVE-2011-0411.

          A rough estimate of what this requires:
          - A new main.cf parameter and documentation.
          - A way to turn off buffering on VSTREAMs. Alternative: use low-level
          read system calls and re-invent VSTREAM timeout/etc. error handling.
          Either this may take a dozen or so lines of code.
          - A new postscreen code module with generic hook to prepend proxies.
          - An event-driven loop that reads the proxy command one character
          at a time until <CR><LF>, length exceeded, EOF, time limit, or
          other error. Another dozen lines.
          - A proxy command parser that does all the necessary sanity checks
          (valid address syntax, numerical TCP ports in the range 1..65535),
          no missing or extra fields. Another dozen lines.
          - Reuse the existing postscreen data structures that are now filled
          with endpoint information from getpeername() and getsockname().

          Wietse
        • Viktor Dukhovni
          ... I concur. Postscreen is only needed in front of public MX hosts to reduce hogging of the SMTP service by ratware. On public MX hosts, you can use a proxy
          Message 4 of 17 , May 27, 2012
          • 0 Attachment
            On Sun, May 27, 2012 at 12:21:49PM +0200, DTNX Postmaster wrote:

            > Also, why would you need a proxy? Unlike HTTP, SMTP does not benefit
            > from front-end caching and the like. HTTP is a stateless, SMTP a
            > stateful protocol? It is easily made highly available, just based on
            > the MX records, and you can implement basic load balancing by having
            > several MX records with the same priority, IIRC.

            I concur. Postscreen is only needed in front of public MX hosts to
            reduce hogging of the SMTP service by ratware. On public MX hosts,
            you can use a proxy that sits in the network path between the
            outside world and the MX hosts, in which case the proxy will not
            rewrite the source IP and no XCLIENT is required.

            Or you can use a proxy like F5 that can implement XCLIENT (the F5
            can perform an programmable initial chat-script before handing
            the client stream to the server).

            Finally, no postscreen is needed in front of submission servers.

            So you can choose proxies that don't rewrite the layer 3 IP address,
            proxies that do, but can do XCLIENT. Or multiple MX hosts with no
            proxies at all:

            example.com. IN MX 0 mx1.example.com.
            example.com. IN MX 0 mx2.example.com.
            example.com. IN MX 0 mx3.example.com.
            example.com. IN MX 0 mx4.example.com.
            ;
            mx1.example.com. IN A 192.0.2.1
            mx1.example.com. IN A 192.0.2.2
            mx1.example.com. IN A 192.0.2.3
            mx1.example.com. IN A 192.0.2.4
            ;
            mx2.example.com. IN A 192.0.2.5
            mx2.example.com. IN A 192.0.2.6
            mx2.example.com. IN A 192.0.2.7
            mx2.example.com. IN A 192.0.2.8
            ;
            mx3.example.com. IN A 192.0.2.9
            mx3.example.com. IN A 192.0.2.10
            mx3.example.com. IN A 192.0.2.11
            mx3.example.com. IN A 192.0.2.12
            ;
            mx4.example.com. IN A 192.0.2.13
            mx4.example.com. IN A 192.0.2.14
            mx4.example.com. IN A 192.0.2.15
            mx4.example.com. IN A 192.0.2.16

            The above gets you 16 MX hosts with no load balancers required.
            You only need load balancers when you start to get to the size
            of Google, Hotmail, ... and they use DNS load-balancers, that
            return geo-proximate IPs for the MX host or any-cast IPs. There
            is likely a second layer of load-balancing below the DNS layer
            at that scale, but very few sites need either.

            --
            Viktor.
          • Wietse Venema
            ... This depends on the proxy. - If it s a packet-level proxy then it just forwards unmodified packets that belong to the same session to the same back-end
            Message 5 of 17 , May 27, 2012
            • 0 Attachment
              Viktor Dukhovni:
              > On Sun, May 27, 2012 at 12:21:49PM +0200, DTNX Postmaster wrote:
              >
              > > Also, why would you need a proxy? Unlike HTTP, SMTP does not benefit
              > > from front-end caching and the like. HTTP is a stateless, SMTP a
              > > stateful protocol? It is easily made highly available, just based on
              > > the MX records, and you can implement basic load balancing by having
              > > several MX records with the same priority, IIRC.
              >
              > I concur. Postscreen is only needed in front of public MX hosts to
              > reduce hogging of the SMTP service by ratware. On public MX hosts,
              > you can use a proxy that sits in the network path between the
              > outside world and the MX hosts, in which case the proxy will not
              > rewrite the source IP and no XCLIENT is required.

              This depends on the proxy.

              - If it's a packet-level proxy then it just forwards unmodified
              packets that belong to the same session to the same back-end SMTP
              server.

              - If it's a circuit-level proxy like nginx, then it needs to forward
              session info in-band, with XCLIENT or equivalent.

              Circuit-level proxies can do a few things that are difficult with
              packet-level proxies, such as sitting in a remote network, or
              that are impossible such as off-loading TLS or AUTH.

              [fan-out with four MX records and four a records per MX name]

              > The above gets you 16 MX hosts with no load balancers required.
              > You only need load balancers when you start to get to the size
              > of Google, Hotmail, ... and they use DNS load-balancers, that
              > return geo-proximate IPs for the MX host or any-cast IPs. There
              > is likely a second layer of load-balancing below the DNS layer
              > at that scale, but very few sites need either.

              Some people don't understand the difference between browser-to-server
              HTTP, and MTA-to-MTA SMTP.

              Wietse
            • Wietse Venema
              ... If smtpd is sufficient for you, then nginx (and other XCLIENT capable proxies, see Victor s post) will do the job. Wietse
              Message 6 of 17 , May 27, 2012
              • 0 Attachment
                Wietse Venema:
                > David Touzeau:
                > > Dear
                > >
                > > We are facing an problem that we cannot resolve...
                > > Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
                > > We have made a discuss with HaProxy founder in order to implement the
                > > XCLIENT protcol but this is difficult for him to implement such protocol.
                >
                > FYI nginx implements XCLIENT (and more). But it does not matter,
                > since postscreen has no proxy support.

                If smtpd is sufficient for you, then nginx (and other XCLIENT
                capable proxies, see Victor's post) will do the job.

                Wietse
              • David Touzeau
                Dear I have played with DNS and MX but the problem is DNS did not care about servers load and if servers have a huge queue. It just balance to next MX only if
                Message 7 of 17 , May 28, 2012
                • 0 Attachment
                  Dear

                  I have played with DNS and MX but the problem is DNS did not care about
                  servers load and if servers have a huge queue.
                  It just balance to next MX only if the server did not respond.
                  Load-balancing is a cool feature that take care on the server health.
                  Especially if you add services on the server such as Spamassassin,
                  backup mails, MDA...

                  Wietse, i copy the founder of HaProxy (Willy Tarreau) in this mail in
                  order to introduce him and see if he can help you implementing such
                  protocol in PostScreen/smtpd.

                  Best regards.





                  Le 27/05/2012 15:25, Wietse Venema a écrit :
                  > David Touzeau:
                  >> Dear
                  >>
                  >> We are facing an problem that we cannot resolve...
                  >> Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
                  >> We have made a discuss with HaProxy founder in order to implement the
                  >> XCLIENT protcol but this is difficult for him to implement such protocol.
                  > FYI nginx implements XCLIENT (and more). But it does not matter,
                  > since postscreen has no proxy support.
                  >
                  >> In other way it seems that PostScreen is not really compatible with the
                  >> proxy protocol.
                  > First, there needs to be a configuration parameter that tells
                  > postscreen what kind of proxy command will be prepended to the
                  > client's SMTP stream.
                  >
                  > /etc/postfix/main.cf:
                  > # Is there a better name than "proxy protocol"?
                  > postscreen_proxy_protocol = whatever
                  >
                  > With this, postscreen will drop connections that fail to conform
                  > to the configured protocol.
                  >
                  > Regardless of command format details, if the proxy prepends a command
                  > to the client's SMTP stream, then postscreen must use unbuffered
                  > I/O to read that command. If buffering were turned on, the buffering
                  > layer could read past the proxy's<CR><LF> and eat up part of the
                  > client input kind-of like CVE-2011-0411.
                  >
                  > A rough estimate of what this requires:
                  > - A new main.cf parameter and documentation.
                  > - A way to turn off buffering on VSTREAMs. Alternative: use low-level
                  > read system calls and re-invent VSTREAM timeout/etc. error handling.
                  > Either this may take a dozen or so lines of code.
                  > - A new postscreen code module with generic hook to prepend proxies.
                  > - An event-driven loop that reads the proxy command one character
                  > at a time until<CR><LF>, length exceeded, EOF, time limit, or
                  > other error. Another dozen lines.
                  > - A proxy command parser that does all the necessary sanity checks
                  > (valid address syntax, numerical TCP ports in the range 1..65535),
                  > no missing or extra fields. Another dozen lines.
                  > - Reuse the existing postscreen data structures that are now filled
                  > with endpoint information from getpeername() and getsockname().
                  >
                  > Wietse
                  >
                  >
                • DTNX Postmaster
                  ... Did you read (and understand) Victor s example? Did you test with multiple servers that had the same preference in their MX record? If they do not have the
                  Message 8 of 17 , May 28, 2012
                  • 0 Attachment
                    On May 28, 2012, at 10:26, David Touzeau wrote:

                    > I have played with DNS and MX but the problem is DNS did not care about servers load and if servers have a huge queue.
                    > It just balance to next MX only if the server did not respond.
                    > Load-balancing is a cool feature that take care on the server health.
                    > Especially if you add services on the server such as Spamassassin, backup mails, MDA...

                    Did you read (and understand) Victor's example?

                    Did you test with multiple servers that had the same preference in
                    their MX record? If they do not have the same preference, the server
                    with the lowest preference value will always be preferred, and failover
                    to the others will indeed only happen when that server is down.

                    --

                    If you operate at scale, you build out your SMTP infrastructure not
                    just in width, but also in depth;

                    1) Mail exchangers. Your first 'line of defense'. These show up in your
                    MX records, and run postscreen to fend off bot traffic and the like.
                    They reject everything you know you're not going to accept, BEFORE the
                    queue. How much depends on your profile, but in our case at more than
                    70% of connections never makes it to the queue. Everything that is
                    accepted is passed on immediately.

                    These do not have any local user mailboxes, and do not do any
                    significant amount of content filtering. They should never have
                    significant amounts of mail in the queue.

                    2) Mail routing. Where does this message go? Does it need to be
                    archived, copied, rerouted?

                    3) Content filtering. This is where Spamassassin runs, for example.
                    Spam is marked, everything is passed on. Nothing is bounced, ever.

                    4) Mailbox server. Accepts the message and sorts them into local
                    folders, where they are stored on disk. This is where POP3/IMAP daemons
                    run to serve users accessing their mailboxes.

                    5) Submission server. This is what your clients talk to to send their
                    outgoing mail. Requires authentication, runs on port 587, and does not
                    need postscreen. May do some basic checks, but passes messages on as
                    quickly as possible. Should never have much of a queue.

                    6) Relay servers. These are the servers that talk to the rest of the
                    internet, for outgoing mail. Since you have little control over the
                    availability and load of the servers you are communicating with, mail
                    may be in the queue here for a while, up to several days.

                    --

                    In most setups, several of these roles can be combined; MX and routing
                    generally go together, and can take care of outgoing relay as well.
                    Content filtering and submission can run on the mailbox servers.

                    But really, find the real bottleneck. You should not need to load
                    balance postscreen, and it's quite likely that it is not the right
                    place to start optimizing.

                    Where are you maxing out? CPU, memory, disk? Is it a single customer
                    that can be split off to a seperate box, perhaps?

                    Cya,
                    Jona

                    --

                    > Le 27/05/2012 15:25, Wietse Venema a écrit :
                    >> David Touzeau:
                    >>> Dear
                    >>>
                    >>> We are facing an problem that we cannot resolve...
                    >>> Our main goal is to implement a load-balancer in front of Postfix (HaProxy).
                    >>> We have made a discuss with HaProxy founder in order to implement the
                    >>> XCLIENT protcol but this is difficult for him to implement such protocol.
                    >> FYI nginx implements XCLIENT (and more). But it does not matter,
                    >> since postscreen has no proxy support.
                    >>
                    >>> In other way it seems that PostScreen is not really compatible with the
                    >>> proxy protocol.
                    >> First, there needs to be a configuration parameter that tells
                    >> postscreen what kind of proxy command will be prepended to the
                    >> client's SMTP stream.
                    >>
                    >> /etc/postfix/main.cf:
                    >> # Is there a better name than "proxy protocol"?
                    >> postscreen_proxy_protocol = whatever
                    >>
                    >> With this, postscreen will drop connections that fail to conform
                    >> to the configured protocol.
                    >>
                    >> Regardless of command format details, if the proxy prepends a command
                    >> to the client's SMTP stream, then postscreen must use unbuffered
                    >> I/O to read that command. If buffering were turned on, the buffering
                    >> layer could read past the proxy's<CR><LF> and eat up part of the
                    >> client input kind-of like CVE-2011-0411.
                    >>
                    >> A rough estimate of what this requires:
                    >> - A new main.cf parameter and documentation.
                    >> - A way to turn off buffering on VSTREAMs. Alternative: use low-level
                    >> read system calls and re-invent VSTREAM timeout/etc. error handling.
                    >> Either this may take a dozen or so lines of code.
                    >> - A new postscreen code module with generic hook to prepend proxies.
                    >> - An event-driven loop that reads the proxy command one character
                    >> at a time until<CR><LF>, length exceeded, EOF, time limit, or
                    >> other error. Another dozen lines.
                    >> - A proxy command parser that does all the necessary sanity checks
                    >> (valid address syntax, numerical TCP ports in the range 1..65535),
                    >> no missing or extra fields. Another dozen lines.
                    >> - Reuse the existing postscreen data structures that are now filled
                    >> with endpoint information from getpeername() and getsockname().
                    >>
                    >> Wietse
                    >>
                    >>
                  • Wietse Venema
                    ... First, just like SMTP and HTTP protocol documentation, HAPROXY documentation states nowhere that any particular information must be sent (or received) in
                    Message 9 of 17 , May 29, 2012
                    • 0 Attachment
                      Willy Tarreau:
                      > > >Regardless of command format details, if the proxy prepends a command
                      > > >to the client's SMTP stream, then postscreen must use unbuffered
                      > > >I/O to read that command. If buffering were turned on, the buffering
                      > > >layer could read past the proxy's<CR><LF> and eat up part of the
                      > > >client input kind-of like CVE-2011-0411.
                      >
                      > Precisely on this point there is an easier way, it consists in using
                      > recv(MSG_PEEK). The big advantage is that you don't need to store the
                      > temporary bytes you've read since they remain in the kernel's buffers.
                      > So it more or less looks like this :

                      First, just like SMTP and HTTP protocol documentation, HAPROXY
                      documentation states nowhere that any particular information must
                      be sent (or received) in exactly one TCP segment.

                      If this atomicity is an essential requirement of the HAPROXY protocol,
                      then that had better be made explicit in the documentation.

                      Second, it makes little sense to re-invent all the error and
                      time-limit handling that Postfix already has. I prefer to reuse the
                      line reading routine that postscreen already has, instead of reaching
                      for the lowest-level kernel API.

                      Wietse
                    • Wietse Venema
                      ... Postscreen, faced with the same non-problem in SMTP, does exactly the same thing. The line read routine would need to be moved out of the dummy SMTP engine
                      Message 10 of 17 , May 29, 2012
                      • 0 Attachment
                        Willy Tarreau:
                        > Hi Wietse,
                        >
                        > On Tue, May 29, 2012 at 08:18:35AM -0400, Wietse Venema wrote:
                        > > Willy Tarreau:
                        > > > > >Regardless of command format details, if the proxy prepends a command
                        > > > > >to the client's SMTP stream, then postscreen must use unbuffered
                        > > > > >I/O to read that command. If buffering were turned on, the buffering
                        > > > > >layer could read past the proxy's<CR><LF> and eat up part of the
                        > > > > >client input kind-of like CVE-2011-0411.
                        > > >
                        > > > Precisely on this point there is an easier way, it consists in using
                        > > > recv(MSG_PEEK). The big advantage is that you don't need to store the
                        > > > temporary bytes you've read since they remain in the kernel's buffers.
                        > > > So it more or less looks like this :
                        > >
                        > > First, just like SMTP and HTTP protocol documentation, HAPROXY
                        > > documentation states nowhere that any particular information must
                        > > be sent (or received) in exactly one TCP segment.
                        >
                        > No, there is no such requirement, as this can never be guaranteed anywhere.
                        > That's why in my example, there was a return on incomplete lines, waiting
                        > for the next event to try to complete the line.

                        Postscreen, faced with the same non-problem in SMTP, does exactly
                        the same thing. The line read routine would need to be moved out
                        of the dummy SMTP engine so that it can be reused to read proxy
                        data, whether from haproxy, xclient or something else.

                        Wietse
                      • Wietse Venema
                        Adding a haproxy-to-postscreen adapter turns out to be pretty trivial. However, a major code rewrite would be needed in the way that postscreen(8) talks to
                        Message 11 of 17 , May 31, 2012
                        • 0 Attachment
                          Adding a haproxy-to-postscreen adapter turns out to be pretty
                          trivial. However, a major code rewrite would be needed in the way
                          that postscreen(8) talks to smtpd(8).

                          Begin background:

                          Postscreen is optimized for the following case: a client connects,
                          postscreen looks up the client IP address in its temporary cache,
                          and when the client is "OK", postscreen sends the connection's file
                          descriptor to a Postfix SMTP server process and gets out of the loop.

                          There is no other communication between postscreen and SMTP server
                          processes. The file descriptor carries all information that an SMTP
                          server process needs. In fact, the file descriptor is indistinguishable
                          from a file descriptor that an SMTP server gets when it is configured
                          in master.cf to listen directly on the SMTP port.

                          End background.

                          To make postscreen work with before-postscreen proxies, it either
                          has to become a proxy itself (over my dead body) or Postfix needs
                          a small protocol to send (attributes plus a file descriptor) from
                          postscreen to smtpd.

                          It's relatively simple to pass a few attributes from haproxy to
                          postscreen with a simple hard-coded non-reusable protocol. On the
                          other hand, Postfix support for (file descriptor + arbitrary attribute
                          passing) will have to be reusable (*), so that the same infrastructure
                          can be used later to improve Postfix. For example, to hand off a
                          connection mid-session from postscreen to smtpd, something that is
                          currently not possible.

                          Wietse

                          (*) The client decides what attributes to send and passes the
                          attributes + file descriptor to the low-level sender infrastructure;
                          the low-level receiver infrastructure first reads the attributes
                          into a hash and then passes the attributes and file descriptor up
                          to the application, in an application-specified order, and deals
                          with missing attributes and other problems.
                        • Wietse Venema
                          ... Sorry, that might work for haproxy, but could complicate attempts to add support for other protocols. If the proxy talks to postscreen then that is where
                          Message 12 of 17 , Jun 1, 2012
                          • 0 Attachment
                            Willy Tarreau:
                            > I'm totally in sync with you on this. On the one hand, if it is as
                            > trivial to make smtpd parse the PROXY line as it was for postscreen,
                            > it can solve the problem by having postscreen not consume the first
                            > line, which makes sense in that postscreen remains the first layer
                            > analyser which doesn't mangle data on the connection. But I can also
                            > understand that you're not necessarily interested in supporting a new
                            > method of passing connection information.

                            Sorry, that might work for haproxy, but could complicate attempts
                            to add support for other protocols. If the proxy talks to postscreen
                            then that is where its protocol will terminate.

                            Wietse
                          • Wietse Venema
                            ... In an event-driven program such as postscreen, this code breaks when the proxy line arrives as multiple fragments. If the program does not drain proxy line
                            Message 13 of 17 , Jun 7, 2012
                            • 0 Attachment
                              Willy Tarreau:
                              > > >Regardless of command format details, if the proxy prepends a command
                              > > >to the client's SMTP stream, then postscreen must use unbuffered
                              > > >I/O to read that command. If buffering were turned on, the buffering
                              > > >layer could read past the proxy's<CR><LF> and eat up part of the
                              > > >client input kind-of like CVE-2011-0411.
                              >
                              > Precisely on this point there is an easier way, it consists in using
                              > recv(MSG_PEEK). The big advantage is that you don't need to store the
                              > temporary bytes you've read since they remain in the kernel's buffers.
                              > So it more or less looks like this :
                              >
                              > len = recv(fd, trash, sizeof(trash), MSG_PEEK);
                              > if (len == -1 && errno == EAGAIN)
                              > return;
                              >
                              > lf = memchr(trash, '\n', len);
                              > if (lf == NULL) {
                              > if (len < trash) /* Huh?? */
                              > return;
                              > /* else abort the connection */
                              > }

                              In an event-driven program such as postscreen, this code breaks
                              when the proxy line arrives as multiple fragments.

                              If the program does not drain proxy line fragments from the kernel
                              buffer, then the socket remains readable and the program will go
                              into a read-notification loop until the entire line is received.

                              This implies that the following suggestion is not valid for an
                              event-driven program such as postscreen:

                              > On the one hand, if it is as trivial to make smtpd parse the PROXY
                              > line as it was for postscreen, it can solve the problem by having
                              > postscreen not consume the first line, which makes sense in that
                              > postscreen remains the first layer analyser which doesn't mangle
                              > data on the connection.

                              Either you need to update the protocol spec (require non-fragmented
                              proxy lines) or provide a code example that doesn't go into a
                              read-notification loop when the proxy line arrives as multiple
                              fragments.

                              Wietse
                            • Willy Tarreau
                              Hi Wietse, [ just subscribed to the list, I realized that our past conversation was dropped since I was not subscribed, never mind ] ... Indeed this is totally
                              Message 14 of 17 , Jun 8, 2012
                              • 0 Attachment
                                Hi Wietse,

                                [ just subscribed to the list, I realized that our past conversation
                                was dropped since I was not subscribed, never mind ]

                                On Thu, Jun 07, 2012 at 08:16:53PM -0400, Wietse Venema wrote:
                                > Willy Tarreau:
                                > > > >Regardless of command format details, if the proxy prepends a command
                                > > > >to the client's SMTP stream, then postscreen must use unbuffered
                                > > > >I/O to read that command. If buffering were turned on, the buffering
                                > > > >layer could read past the proxy's<CR><LF> and eat up part of the
                                > > > >client input kind-of like CVE-2011-0411.
                                > >
                                > > Precisely on this point there is an easier way, it consists in using
                                > > recv(MSG_PEEK). The big advantage is that you don't need to store the
                                > > temporary bytes you've read since they remain in the kernel's buffers.
                                > > So it more or less looks like this :
                                > >
                                > > len = recv(fd, trash, sizeof(trash), MSG_PEEK);
                                > > if (len == -1 && errno == EAGAIN)
                                > > return;
                                > >
                                > > lf = memchr(trash, '\n', len);
                                > > if (lf == NULL) {
                                > > if (len < trash) /* Huh?? */
                                > > return;
                                > > /* else abort the connection */
                                > > }
                                >
                                > In an event-driven program such as postscreen, this code breaks
                                > when the proxy line arrives as multiple fragments.
                                >
                                > If the program does not drain proxy line fragments from the kernel
                                > buffer, then the socket remains readable and the program will go
                                > into a read-notification loop until the entire line is received.

                                Indeed this is totally true. I would say that given the short size
                                of the message the risk of this occurring is barely 0 explaining
                                whit this has probably never hit anybody, but it would be sufficient
                                that someone implements the protocol using a series of
                                write(fd, buf++, 1) and you'd be spinning (even worse if it dies in
                                the middle).

                                I've seen implementations doing recv(MSG_PEEK) and reject connections
                                from incomplete messages (again, almost zero risk but YMMV).

                                In haproxy, I'm using the input buffer associated to the fd, so I
                                don't have this problem. I'm basically doing a recv() on the buffer.
                                I think it is equivalent to the VSTREAM you're using.

                                In postscreen you don't want to do this since you don't want to
                                consume any possible incoming data (btw you probably drop the
                                connection if you get any data at this point). That said, if you
                                have a buffer associated to the connection, then you can perform
                                the first MSG_PEEK to check the pending data size and then a real
                                recv() to only consume up to the end of line.

                                But then doing so indeed invalidates the following suggestion.

                                > This implies that the following suggestion is not valid for an
                                > event-driven program such as postscreen:
                                >
                                > > On the one hand, if it is as trivial to make smtpd parse the PROXY
                                > > line as it was for postscreen, it can solve the problem by having
                                > > postscreen not consume the first line, which makes sense in that
                                > > postscreen remains the first layer analyser which doesn't mangle
                                > > data on the connection.


                                > Either you need to update the protocol spec (require non-fragmented
                                > proxy lines)

                                I have mixed opinions on this. On the one hand, we can't really impose
                                lower layers segmentation behaviour, so from a layering perspective, it
                                is not correct. On the other hand, the use cases for the protocol are
                                very specific. We're the very first segment over the connection so we
                                are always allowed to send at least one MSS. Nobody should sanely use
                                this proxy line on connections with an MSS lower than the 116 bytes a
                                max line may be for long IPv6 addresses and ports.

                                So indeed, I'm tempted to follow your suggestion, it will ease processing
                                for everyone and ensure that nobody tries sending fragmented lines. We'd
                                rely on a sane lower layer and declare other cases out of scope.

                                > or provide a code example that doesn't go into a
                                > read-notification loop when the proxy line arrives as multiple
                                > fragments.

                                With a buffer this problem does not happen, but it's the first case I'm
                                facing this need with fd passing, which makes me scratch my head a lot.
                                I really like the way you're plugging postscreen in front of smtpd, and
                                I'd like to ensure we don't make it complex to keep this nice model.

                                That's why I think that adding a sane requirement in the spec should be
                                the most adequate solution. If in the mean time you get a smarter idea,
                                do not hesitate to share it :-)

                                I'll keep thinking about it a bit before updating it. I think I will also
                                propose some generic code in the spec for both sides.

                                Best regards,
                                Willy
                              • Wietse Venema
                                Non-production release postfix-2.10-20120617-nonprod has support for up-stream proxy agents in postscreen(8) and smtpd(8). To enable, specify one of:
                                Message 15 of 17 , Jun 17, 2012
                                • 0 Attachment
                                  Non-production release postfix-2.10-20120617-nonprod has support
                                  for up-stream proxy agents in postscreen(8) and smtpd(8).

                                  To enable, specify one of:

                                  postscreen_upstream_proxy_protocol = haproxy
                                  smtpd_upstream_proxy_protocol = haproxy

                                  haproxy is not the only proxy agent that works with Postfix. Support
                                  for nginx with proxied SASL authentication is available in Postfix
                                  2.9 smtpd(8). This uses the XCLIENT protocol.

                                  Wietse
                                • Willy Tarreau
                                  Hi Wietse, ... This is awesome work, it opens a wider interoperability between our components which are commonly placed close to each other in a number of
                                  Message 16 of 17 , Jun 17, 2012
                                  • 0 Attachment
                                    Hi Wietse,

                                    On Sun, Jun 17, 2012 at 08:25:12PM -0400, Wietse Venema wrote:
                                    > Non-production release postfix-2.10-20120617-nonprod has support
                                    > for up-stream proxy agents in postscreen(8) and smtpd(8).
                                    >
                                    > To enable, specify one of:
                                    >
                                    > postscreen_upstream_proxy_protocol = haproxy
                                    > smtpd_upstream_proxy_protocol = haproxy
                                    >
                                    > haproxy is not the only proxy agent that works with Postfix. Support
                                    > for nginx with proxied SASL authentication is available in Postfix
                                    > 2.9 smtpd(8). This uses the XCLIENT protocol.

                                    This is awesome work, it opens a wider interoperability between our
                                    components which are commonly placed close to each other in a number
                                    of infrastructures.

                                    I hope David (who originally asked for the feature) will quickly test
                                    and provide us with some feedback.

                                    I'll forward your mail to the haproxy mailing list, since I know there
                                    are a number of postfix users there too.

                                    Best regards,
                                    Willy
                                  • Wietse Venema
                                    ... After another week of testing, this is now released as a regular development release postfix-2.10-20120625. Wietse
                                    Message 17 of 17 , Jun 25, 2012
                                    • 0 Attachment
                                      > Non-production release postfix-2.10-20120617-nonprod has support
                                      > for up-stream proxy agents in postscreen(8) and smtpd(8).
                                      >
                                      > To enable, specify one of:
                                      >
                                      > postscreen_upstream_proxy_protocol = haproxy
                                      > smtpd_upstream_proxy_protocol = haproxy
                                      >
                                      > haproxy is not the only proxy agent that works with Postfix. Support
                                      > for nginx with proxied SASL authentication is available in Postfix
                                      > 2.9 smtpd(8). This uses the XCLIENT protocol.

                                      After another week of testing, this is now released as a regular
                                      development release postfix-2.10-20120625.

                                      Wietse
                                    Your message has been successfully submitted and would be delivered to recipients shortly.