Loading ...
Sorry, an error occurred while loading the content.

Re: SMTP proxy????

Expand Messages
  • Victor Duchovni
    ... Well, the proxy won t know what to do before the user authenticates, and you say the the authentication databases are split, so it is far from clear how
    Message 1 of 12 , Feb 1, 2011
    • 0 Attachment
      On Tue, Feb 01, 2011 at 05:33:14PM +0100, Ignacio Garcia wrote:

      > Hi there. Hi, I've been googling around all morning and I'm
      > completely ignorant on what I'm going to ask, so please forgive me if I
      > make no sense. I have 2 independent servers running
      > postifx+mysql+(other_things) all controlled from a nice web interfacce
      > called ISPConfig3. Those 2 servers are completely independent with many
      > domains configured in each of them. Authentication is done against each
      > server's separate and different mysql database. I'm testing Perdition
      > for imap and pop3 connections so webmail access is more
      > consistent/unified, and in case of customers with email services in both
      > servers, we make it easier for them since the proxy redirects
      > connections to the right imap server. My question: is there such a
      > similar product (SMTP proxy) that can be configured in the same way to
      > hide the real smtp servers and
      > deliver/accept_mail_from_our_2_different_pools_of_users using the
      > correct server?

      Well, the proxy won't know what to do before the user authenticates,
      and you say the the authentication databases are split, so it is far
      from clear how you expect this could work.

      However, if Perdition presents a unified IMAP interface, you could
      perhaps use an "rimap" backend with Cyrus SASL to authenticate the
      user.

      I am not aware of any SMTP proxies whose downstream SMTP server is
      selected after user authentication. It is probably easiest to just
      operate a unified submission server that authenticates the union of the
      two sets of users, and then routes to the right server via sender-based
      routing. In other-words, not a proxy but a store-and-forward MSA.

      Postfix can do that.

      --
      Viktor.
    • Frank Bonnet
      ... There is a mysql proxy software , maybe it could help to authenticate users from both databases ? http://forge.mysql.com/wiki/MySQL_Proxy
      Message 2 of 12 , Feb 1, 2011
      • 0 Attachment
        On 02/01/2011 05:43 PM, Victor Duchovni wrote:
        > On Tue, Feb 01, 2011 at 05:33:14PM +0100, Ignacio Garcia wrote:
        >
        >> Hi there. Hi, I've been googling around all morning and I'm
        >> completely ignorant on what I'm going to ask, so please forgive me if I
        >> make no sense. I have 2 independent servers running
        >> postifx+mysql+(other_things) all controlled from a nice web interfacce
        >> called ISPConfig3. Those 2 servers are completely independent with many
        >> domains configured in each of them. Authentication is done against each
        >> server's separate and different mysql database. I'm testing Perdition
        >> for imap and pop3 connections so webmail access is more
        >> consistent/unified, and in case of customers with email services in both
        >> servers, we make it easier for them since the proxy redirects
        >> connections to the right imap server. My question: is there such a
        >> similar product (SMTP proxy) that can be configured in the same way to
        >> hide the real smtp servers and
        >> deliver/accept_mail_from_our_2_different_pools_of_users using the
        >> correct server?
        > Well, the proxy won't know what to do before the user authenticates,
        > and you say the the authentication databases are split, so it is far
        > from clear how you expect this could work.
        >
        > However, if Perdition presents a unified IMAP interface, you could
        > perhaps use an "rimap" backend with Cyrus SASL to authenticate the
        > user.
        >
        > I am not aware of any SMTP proxies whose downstream SMTP server is
        > selected after user authentication. It is probably easiest to just
        > operate a unified submission server that authenticates the union of the
        > two sets of users, and then routes to the right server via sender-based
        > routing. In other-words, not a proxy but a store-and-forward MSA.
        >
        > Postfix can do that.
        >
        There is a mysql proxy software , maybe it could help to authenticate
        users from both databases ?

        http://forge.mysql.com/wiki/MySQL_Proxy
      • Reindl Harald
        ... What about using only one mysql-server and the other one as replication slave, postfix is readonly So you would have the backend which writes to the master
        Message 3 of 12 , Feb 1, 2011
        • 0 Attachment
          Am 01.02.2011 17:51, schrieb Frank Bonnet:
          > On 02/01/2011 05:43 PM, Victor Duchovni wrote:
          >> On Tue, Feb 01, 2011 at 05:33:14PM +0100, Ignacio Garcia wrote:
          >>
          >>> Hi there. Hi, I've been googling around all morning and I'm
          >>> completely ignorant on what I'm going to ask, so please forgive me if I
          >>> make no sense. I have 2 independent servers running
          >>> postifx+mysql+(other_things) all controlled from a nice web interfacce
          >>> called ISPConfig3. Those 2 servers are completely independent with many
          >>> domains configured in each of them. Authentication is done against each
          >>> server's separate and different mysql database. I'm testing Perdition
          >>> for imap and pop3 connections so webmail access is more
          >>> consistent/unified, and in case of customers with email services in both
          >>> servers, we make it easier for them since the proxy redirects
          >>> connections to the right imap server. My question: is there such a
          >>> similar product (SMTP proxy) that can be configured in the same way to
          >>> hide the real smtp servers and
          >>> deliver/accept_mail_from_our_2_different_pools_of_users using the
          >>> correct server?
          >> Well, the proxy won't know what to do before the user authenticates,
          >> and you say the the authentication databases are split, so it is far
          >> from clear how you expect this could work.
          >>
          >> However, if Perdition presents a unified IMAP interface, you could
          >> perhaps use an "rimap" backend with Cyrus SASL to authenticate the
          >> user.
          >>
          >> I am not aware of any SMTP proxies whose downstream SMTP server is
          >> selected after user authentication. It is probably easiest to just
          >> operate a unified submission server that authenticates the union of the
          >> two sets of users, and then routes to the right server via sender-based
          >> routing. In other-words, not a proxy but a store-and-forward MSA.
          >>
          >> Postfix can do that.
          >>
          > There is a mysql proxy software , maybe it could help to authenticate
          > users from both databases ?
          >
          > http://forge.mysql.com/wiki/MySQL_Proxy

          What about using only one mysql-server and the other one as
          replication slave, postfix is readonly

          So you would have the backend which writes to the master
          and the load from the smtp-servers is spread and as
          benefit if the master dies there is a additional backup
        • Ignacio Garcia
          On Tue, 1 Feb 2011 11:43:25 -0500, Victor Duchovni ... Victor, thanks for your quick response. yes, I did not take into account that authentication does not
          Message 4 of 12 , Feb 1, 2011
          • 0 Attachment
            On Tue, 1 Feb 2011 11:43:25 -0500, Victor Duchovni
            <Victor.Duchovni@...> wrote:
            > On Tue, Feb 01, 2011 at 05:33:14PM +0100, Ignacio Garcia wrote:
            >
            >> Hi there. Hi, I've been googling around all morning and I'm
            >> completely ignorant on what I'm going to ask, so please forgive me if I
            >> make no sense. I have 2 independent servers running
            >> postifx+mysql+(other_things) all controlled from a nice web interfacce
            >> called ISPConfig3. Those 2 servers are completely independent with many
            >> domains configured in each of them. Authentication is done against each
            >> server's separate and different mysql database. I'm testing Perdition
            >> for imap and pop3 connections so webmail access is more
            >> consistent/unified, and in case of customers with email services in both
            >> servers, we make it easier for them since the proxy redirects
            >> connections to the right imap server. My question: is there such a
            >> similar product (SMTP proxy) that can be configured in the same way to
            >> hide the real smtp servers and
            >> deliver/accept_mail_from_our_2_different_pools_of_users using the
            >> correct server?
            >
            > Well, the proxy won't know what to do before the user authenticates,
            > and you say the the authentication databases are split, so it is far
            > from clear how you expect this could work.
            >
            > However, if Perdition presents a unified IMAP interface, you could
            > perhaps use an "rimap" backend with Cyrus SASL to authenticate the
            > user.
            >
            > I am not aware of any SMTP proxies whose downstream SMTP server is
            > selected after user authentication. It is probably easiest to just
            > operate a unified submission server that authenticates the union of the
            > two sets of users, and then routes to the right server via sender-based
            > routing. In other-words, not a proxy but a store-and-forward MSA.
            >
            > Postfix can do that.

            Victor, thanks for your quick response.

            yes, I did not take into account that authentication does not always
            take place in SMTP. So I guess that leaves me with no other option but
            to consider a round robin setup. However, I'm not sure how this works.
            Do I need to setup each postfix server to accept messages from/to both
            sets of users, or in this scenario, if the first connection fails, it'll
            try the second automagically?

            Can anybody point me to a tutorail, howto, etc. on how to setup postfix
            in a round robin environment?

            Thanks so much.

            Ignacio
          • Victor Duchovni
            ... It does if you require it, as is normal with a submission service. ... No, you can create a unified submission server which accepts submissions for both
            Message 5 of 12 , Feb 1, 2011
            • 0 Attachment
              On Tue, Feb 01, 2011 at 06:11:58PM +0100, Ignacio Garcia wrote:

              > > However, if Perdition presents a unified IMAP interface, you could
              > > perhaps use an "rimap" backend with Cyrus SASL to authenticate the
              > > user.
              > >
              > > I am not aware of any SMTP proxies whose downstream SMTP server is
              > > selected after user authentication. It is probably easiest to just
              > > operate a unified submission server that authenticates the union of the
              > > two sets of users, and then routes to the right server via sender-based
              > > routing. In other-words, not a proxy but a store-and-forward MSA.
              > >
              > > Postfix can do that.
              >
              > Victor, thanks for your quick response.
              >
              > yes, I did not take into account that authentication does not always
              > take place in SMTP.

              It does if you require it, as is normal with a submission service.

              > So I guess that leaves me with no other option but
              > to consider a round robin setup.

              No, you can create a unified submission server which accepts submissions
              for both environments, and forwards to the right server for outbound
              delivery.

              > However, I'm not sure how this works.
              > Do I need to setup each postfix server to accept messages from/to both
              > sets of users, or in this scenario, if the first connection fails, it'll
              > try the second automagically?

              > Can anybody point me to a tutorail, howto, etc. on how to setup postfix
              > in a round robin environment?

              You are not thinking very clearly yet. You must distinguish clearly
              between:

              - Submission, users submitting mail for outgoing delivery. This is
              visible to users, since they set the server in question as their
              MUAs "SMTP server". This needs its own design, and it is what I
              thought you wanted help with.

              - Inbound MX service. Just publish appropriate MX records, and add
              front-end SMTP servers to each of the two environments as necessary.
              In ISP-grade implementations the SMTP inbound servers are not also
              IMAP servers, rather they forward mail to IMAP servers (via LMTP
              or SMTP).

              Which do you need help with? State clear requirements for either or
              each problem, and work from there.

              --
              Viktor.
            • Ignacio Garcia
              ... yes, you re right, sorry. Maybe if I tell you what I want to do I can make myself clearer. We have both a submission service and inbound mx service. we
              Message 6 of 12 , Feb 1, 2011
              • 0 Attachment
                > You are not thinking very clearly yet. You must distinguish clearly
                > between:
                >
                > - Submission, users submitting mail for outgoing delivery. This is
                > visible to users, since they set the server in question as their
                > MUAs "SMTP server". This needs its own design, and it is what I
                > thought you wanted help with.
                >
                > - Inbound MX service. Just publish appropriate MX records, and add
                > front-end SMTP servers to each of the two environments as necessary.
                > In ISP-grade implementations the SMTP inbound servers are not also
                > IMAP servers, rather they forward mail to IMAP servers (via LMTP
                > or SMTP).
                >
                > Which do you need help with? State clear requirements for either or
                > each problem, and work from there.


                yes, you're right, sorry. Maybe if I tell you what I want to do I can
                make myself clearer. We have both a submission service and inbound mx
                service. we want to have a unified smtp.mycompany.com so all
                "submissions" can be processed using this canonical domain name. i
                believe that is not too dificult. We also want to run both servers so 1
                is mx-backup of the other and viceversa. However, I forsee several
                problems here:

                - both servers need to check against both databases for valid
                destinations
                - each server must know if delivery is to be local or it has to be
                relayed to the other server.
                - one server virtual_transport=maildrop (to courier-imap), the other
                =dovecot (we are going with dovecot for the future, the one with courier
                is older, but plenty of users)

                So, is there any hope?

                Ignacio
              • Victor Duchovni
                ... Good if you know how to handle submission, you re half way there... ... This splits into two use-cases: * Relay - Each server accepts mail for the other s
                Message 7 of 12 , Feb 1, 2011
                • 0 Attachment
                  On Tue, Feb 01, 2011 at 06:50:04PM +0100, Ignacio Garcia wrote:

                  > > Which do you need help with? State clear requirements for either or
                  > > each problem, and work from there.
                  >
                  >
                  > yes, you're right, sorry. Maybe if I tell you what I want to do I can
                  > make myself clearer. We have both a submission service and inbound mx
                  > service. we want to have a unified smtp.mycompany.com so all
                  > "submissions" can be processed using this canonical domain name. i
                  > believe that is not too dificult.

                  Good if you know how to handle submission, you're half way there...

                  > We also want to run both servers so 1
                  > is mx-backup of the other and viceversa.

                  This splits into two use-cases:

                  * Relay

                  - Each server accepts mail for the other's domains, and relays
                  them to the right server. All you need is access to a table
                  of valid recipients for the relay domains, and table of
                  said domain names.

                  * Full-service:

                  - Each server accepts mail for all the domains, and delivers
                  to the correct mail store. Need fully unified data model.

                  Which use-case are you aiming for?

                  > - both servers need to check against both databases for valid
                  > destinations

                  Correct (s/destinations/user-addresses/).

                  > - each server must know if delivery is to be local or it has to be
                  > relayed to the other server.

                  Postfix does that automatically. Just add the domain to "relay_domains"
                  and not to virtual_mailbox_domains, or similar.

                  > - one server virtual_transport=maildrop (to courier-imap), the other
                  > =dovecot (we are going with dovecot for the future, the one with courier
                  > is older, but plenty of users)

                  The difference in final delivery mechanisms is immaterial.

                  --
                  Viktor.
                • Ignacio Garcia
                  ... definitely full-service. I do not understand whan you say that I need a fully unified data model. ... ypu re right. ... ok. ... Great!!! So, can you point
                  Message 8 of 12 , Feb 1, 2011
                  • 0 Attachment
                    > This splits into two use-cases:
                    >
                    > * Relay
                    >
                    > - Each server accepts mail for the other's domains, and relays
                    > them to the right server. All you need is access to a table
                    > of valid recipients for the relay domains, and table of
                    > said domain names.
                    >
                    > * Full-service:
                    >
                    > - Each server accepts mail for all the domains, and delivers
                    > to the correct mail store. Need fully unified data model.
                    >
                    > Which use-case are you aiming for?
                    >

                    definitely full-service. I do not understand whan you say that I need a
                    fully unified data model.


                    >> - both servers need to check against both databases for valid
                    >> destinations
                    >
                    > Correct (s/destinations/user-addresses/).

                    ypu're right.

                    >> - each server must know if delivery is to be local or it has to be
                    >> relayed to the other server.
                    >
                    > Postfix does that automatically. Just add the domain to "relay_domains"
                    > and not to virtual_mailbox_domains, or similar.

                    ok.

                    >> - one server virtual_transport=maildrop (to courier-imap), the other
                    >> =dovecot (we are going with dovecot for the future, the one with courier
                    >> is older, but plenty of users)
                    >
                    > The difference in final delivery mechanisms is immaterial.

                    Great!!!


                    So, can you point me to the right direction in learning how to get the
                    full-service configuration done?

                    Thanks!!

                    Ignacio
                  • Victor Duchovni
                    ... In addition to recipient lists, you need mailstore locations, and transport settings, ... to actually deliver the mail. ... This is for the relay
                    Message 9 of 12 , Feb 1, 2011
                    • 0 Attachment
                      On Tue, Feb 01, 2011 at 07:11:52PM +0100, Ignacio Garcia wrote:

                      > > This splits into two use-cases:
                      > >
                      > > * Relay
                      > >
                      > > - Each server accepts mail for the other's domains, and relays
                      > > them to the right server. All you need is access to a table
                      > > of valid recipients for the relay domains, and table of
                      > > said domain names.
                      > >
                      > > * Full-service:
                      > >
                      > > - Each server accepts mail for all the domains, and delivers
                      > > to the correct mail store. Need fully unified data model.
                      > >
                      > > Which use-case are you aiming for?
                      > >
                      >
                      > definitely full-service. I do not understand whan you say that I need a
                      > fully unified data model.

                      In addition to recipient lists, you need mailstore locations, and
                      transport settings, ... to actually deliver the mail.

                      > >> - both servers need to check against both databases for valid
                      > >> destinations
                      > >
                      > > Correct (s/destinations/user-addresses/).
                      >
                      > ypu're right.
                      >
                      > >> - each server must know if delivery is to be local or it has to be
                      > >> relayed to the other server.
                      > >
                      > > Postfix does that automatically. Just add the domain to "relay_domains"
                      > > and not to virtual_mailbox_domains, or similar.
                      >
                      > ok.

                      This is for the "relay" use-case.

                      > >> - one server virtual_transport=maildrop (to courier-imap), the other
                      > >> =dovecot (we are going with dovecot for the future, the one with courier
                      > >> is older, but plenty of users)
                      > >
                      > > The difference in final delivery mechanisms is immaterial.
                      >
                      > Great!!!

                      Sorry, for the full-service use case, both sets of SMTP servers
                      need to support both delivery mechanisms and choose the correct
                      one for each user.

                      Ideally it would be LMTP for both, and the IMAP servers would handle
                      the delivery logistics.

                      > So, can you point me to the right direction in learning how to get the
                      > full-service configuration done?

                      Learn about virtual_alias rewriting or per-user transports in
                      ADDRESS_REWRITING_README.html. Strongly consider using LMTP for
                      all MTA -> Mail store deliveries, so that the MTA does not need
                      to understand the internal architecture of the IMAP store.

                      --
                      Viktor.
                    • Ignacio Garcia
                      Viktor, thanks very much. I ll be reading about all this and hopefully will be asking some more questions once I have more knowledge of this. Ignacio On Tue, 1
                      Message 10 of 12 , Feb 1, 2011
                      • 0 Attachment
                        Viktor, thanks very much. I'll be reading about all this and hopefully
                        will be asking some more questions once I have more knowledge of this.

                        Ignacio


                        On Tue, 1 Feb 2011 13:20:07 -0500, Victor Duchovni
                        <Victor.Duchovni@...> wrote:
                        > On Tue, Feb 01, 2011 at 07:11:52PM +0100, Ignacio Garcia wrote:
                        >
                        >> > This splits into two use-cases:
                        >> >
                        >> > * Relay
                        >> >
                        >> > - Each server accepts mail for the other's domains, and relays
                        >> > them to the right server. All you need is access to a table
                        >> > of valid recipients for the relay domains, and table of
                        >> > said domain names.
                        >> >
                        >> > * Full-service:
                        >> >
                        >> > - Each server accepts mail for all the domains, and delivers
                        >> > to the correct mail store. Need fully unified data model.
                        >> >
                        >> > Which use-case are you aiming for?
                        >> >
                        >>
                        >> definitely full-service. I do not understand whan you say that I need a
                        >> fully unified data model.
                        >
                        > In addition to recipient lists, you need mailstore locations, and
                        > transport settings, ... to actually deliver the mail.
                        >
                        >> >> - both servers need to check against both databases for valid
                        >> >> destinations
                        >> >
                        >> > Correct (s/destinations/user-addresses/).
                        >>
                        >> ypu're right.
                        >>
                        >> >> - each server must know if delivery is to be local or it has to be
                        >> >> relayed to the other server.
                        >> >
                        >> > Postfix does that automatically. Just add the domain to "relay_domains"
                        >> > and not to virtual_mailbox_domains, or similar.
                        >>
                        >> ok.
                        >
                        > This is for the "relay" use-case.
                        >
                        >> >> - one server virtual_transport=maildrop (to courier-imap), the other
                        >> >> =dovecot (we are going with dovecot for the future, the one with courier
                        >> >> is older, but plenty of users)
                        >> >
                        >> > The difference in final delivery mechanisms is immaterial.
                        >>
                        >> Great!!!
                        >
                        > Sorry, for the full-service use case, both sets of SMTP servers
                        > need to support both delivery mechanisms and choose the correct
                        > one for each user.
                        >
                        > Ideally it would be LMTP for both, and the IMAP servers would handle
                        > the delivery logistics.
                        >
                        >> So, can you point me to the right direction in learning how to get the
                        >> full-service configuration done?
                        >
                        > Learn about virtual_alias rewriting or per-user transports in
                        > ADDRESS_REWRITING_README.html. Strongly consider using LMTP for
                        > all MTA -> Mail store deliveries, so that the MTA does not need
                        > to understand the internal architecture of the IMAP store.

                        --
                        Ignacio Garcia
                        Gerente
                        E.S. Oenus SL
                        Tel: 962 961 017
                        SkypeID: oenus.com
                        http://www.oenus.com
                      • Daniel Bromberg
                        ... Speaking not as a Postfix expert but as a DBA, and speaking more on administrative strategy rather than technically, I would consider merging the two
                        Message 11 of 12 , Feb 1, 2011
                        • 0 Attachment
                          On 2/1/2011 12:50 PM, Ignacio Garcia wrote:
                          >> You are not thinking very clearly yet. You must distinguish clearly
                          >> between:
                          >>
                          >> - Submission, users submitting mail for outgoing delivery. This is
                          >> visible to users, since they set the server in question as their
                          >> MUAs "SMTP server". This needs its own design, and it is what I
                          >> thought you wanted help with.
                          >>
                          >> - Inbound MX service. Just publish appropriate MX records, and add
                          >> front-end SMTP servers to each of the two environments as necessary.
                          >> In ISP-grade implementations the SMTP inbound servers are not also
                          >> IMAP servers, rather they forward mail to IMAP servers (via LMTP
                          >> or SMTP).
                          >>
                          >> Which do you need help with? State clear requirements for either or
                          >> each problem, and work from there.
                          >
                          > yes, you're right, sorry. Maybe if I tell you what I want to do I can
                          > make myself clearer. We have both a submission service and inbound mx
                          > service. we want to have a unified smtp.mycompany.com so all
                          > "submissions" can be processed using this canonical domain name. i
                          > believe that is not too dificult. We also want to run both servers so 1
                          > is mx-backup of the other and viceversa. However, I forsee several
                          > problems here:
                          >
                          > - both servers need to check against both databases for valid
                          > destinations
                          > - each server must know if delivery is to be local or it has to be
                          > relayed to the other server.
                          > - one server virtual_transport=maildrop (to courier-imap), the other
                          > =dovecot (we are going with dovecot for the future, the one with courier
                          > is older, but plenty of users)
                          >
                          > So, is there any hope?
                          >
                          > Ignacio
                          Speaking not as a Postfix expert but as a DBA, and speaking more on
                          administrative strategy rather than technically, I would consider
                          merging the two databases. This is just a guess that the two servers are
                          running under the same organizational umbrella and that the DB's serve a
                          similar purpose for a single organization. Ugliness will grow and grow
                          around accommodating two databases that should be one.

                          The unified DB would have a "server_a_user_table" and a
                          "server_b_user_table" ([much] better yet, a unified table with a server
                          tag column so arbitrary servers can be added ultimately). Then you have
                          the flexibility to distinguish a local user lookup versus a relay lookup
                          on each server by configuring each of transport queries appropriately.
                          (Hint:
                          http://dev.mysql.com/doc/refman/5.0/en/control-flow-functions.html#function_if)

                          Do the painful merging steps now and grow a sensible architecture that
                          can last for years. All kinds of dust will get rooted out in the
                          process. Optimistically your business will only need to add servers. An
                          obvious initial benefit is the simplified backup and monitoring
                          requirements.

                          Then as others have mentioned, the MySQL architecture can be replicated,
                          tuned, and accessed remotely with appropriate security settings. It can
                          even be replicated while live by setting it to read-only during initial
                          replication.

                          I believe efficiency in a replicated setting will still be very high.
                          Mail is very heavily read-oriented for DB's: settings/user migration/new
                          users don't change that much but there are 10^n lookups a day.

                          -Daniel
                        Your message has been successfully submitted and would be delivered to recipients shortly.