Loading ...
Sorry, an error occurred while loading the content.

RE: SMTP sessions

Expand Messages
  • Rocco Scappatura
    Hello, ... Have someone further wideing to provide about this argument? rocsca
    Message 1 of 10 , Feb 1, 2009
    • 0 Attachment
      Hello,

      >> > I have a mail gateway system that consists of several
      >> > Postfix+MySQL+Amavisd-new machines behind a load balancer.
      >> >
      >> > I have defined a balancing policy based on number of SMTP sessions
      >> that
      >> > every server has to manage.
      >>
      >> New connections are given to the server with the fewest connections?
      >
      > Yes.
      >
      >> > But, even if the session is perfectly balanced, I see that the
      >> average
      >> > latency of a message in Postfix queues is too high on some machines
      >> and
      >> > quite zero on other.
      >>
      >> Are the same servers overloaded over a long period of time? And
      > lightly
      >> loaded servers remain lightly loaded?
      >
      > Usually.
      >
      >> What is the critical resource? Disk I/O? CPU? Output concurrency?
      >
      > Indeed, the number of sessions is correctly proportional to the weight I
      > have assigned to each server on balancer. But the load of the CPUs of
      > each machine not. I have watched Disk I/O with vmstat and OS never
      > swaps. I have a good quantity of RAM free.
      >
      > I monitored each machine's parameters using vmstat and what I could have
      > noted I is the number of blocked procs which is often nonzero (from 0 to
      > 3) when the mchine is overloaded.
      >
      > What do you mean for output concurrency?
      >
      > I have raised "maxproc" for amavis-filter to reduce the number of
      > blocked procs.
      >
      >> > What I infer is that every session can be used to devilver/send
      >> > different email messages (other then every message as inerently a
      >> > different size).
      >> >
      >> > It is right my argument or Im wrong in something? If yes, has
      > Postfix
      >> > the control of the number of message that could be manage by each
      >> SMTP
      >> > session?
      >>
      >> Take a look at "qshape", is there a lot of deferred mail on some
      >> systems
      >> and not others? Are you doing recipient validation, or accepting and
      >> bouncing a lot of mail?
      >
      > I constantly have monitored the Postfix queues with qshape, particularly
      > active queue:
      >
      > # watch "perl
      > /usr/local/src/postfix-2.5.2/auxiliary/qshape/qshape.pl -s active| head"
      >
      > I have a reasonably "normal" number of deferred emails (no more than 100
      > messages).
      >
      > Nevertheless, I'm doing recipient validation for each mailbox that I
      > manage and verification on each email of every domain for which I
      > forward messages.
      >
      > I fear that the problem is that for each session I can have an unsettled
      > number of messages sent over that session (It could be happen? If yes,
      > It could be depend on MTA settings?) other then an unsettled size of
      > SMTP traffic (which it determs the latency of messages and it could make
      > congestion of postfix active queue more or less heavy).

      Have someone further wideing to provide about this argument?

      rocsca
    • Rocco Scappatura
      Hi, ... Could someone give me some hint about this issue? TIA, rocsca
      Message 2 of 10 , Feb 3, 2009
      • 0 Attachment
        Hi,

        >>> > I have a mail gateway system that consists of several
        >>> > Postfix+MySQL+Amavisd-new machines behind a load balancer.
        >>> >
        >>> > I have defined a balancing policy based on number of SMTP sessions
        >>> that
        >>> > every server has to manage.
        >>>
        >>> New connections are given to the server with the fewest connections?
        >>
        >> Yes.
        >>
        >>> > But, even if the session is perfectly balanced, I see that the
        >>> average
        >>> > latency of a message in Postfix queues is too high on some machines
        >>> and
        >>> > quite zero on other.
        >>>
        >>> Are the same servers overloaded over a long period of time? And
        >> lightly
        >>> loaded servers remain lightly loaded?
        >>
        >> Usually.
        >>
        >>> What is the critical resource? Disk I/O? CPU? Output concurrency?
        >>
        >> Indeed, the number of sessions is correctly proportional to the weight I
        >> have assigned to each server on balancer. But the load of the CPUs of
        >> each machine not. I have watched Disk I/O with vmstat and OS never
        >> swaps. I have a good quantity of RAM free.
        >>
        >> I monitored each machine's parameters using vmstat and what I could have
        >> noted I is the number of blocked procs which is often nonzero (from 0 to
        >> 3) when the mchine is overloaded.
        >>
        >> What do you mean for output concurrency?
        >>
        >> I have raised "maxproc" for amavis-filter to reduce the number of
        >> blocked procs.
        >>
        >>> > What I infer is that every session can be used to devilver/send
        >>> > different email messages (other then every message as inerently a
        >>> > different size).
        >>> >
        >>> > It is right my argument or Im wrong in something? If yes, has
        >> Postfix
        >>> > the control of the number of message that could be manage by each
        >>> SMTP
        >>> > session?
        >>>
        >>> Take a look at "qshape", is there a lot of deferred mail on some
        >>> systems
        >>> and not others? Are you doing recipient validation, or accepting and
        >>> bouncing a lot of mail?
        >>
        >> I constantly have monitored the Postfix queues with qshape, particularly
        >> active queue:
        >>
        >> # watch "perl
        >> /usr/local/src/postfix-2.5.2/auxiliary/qshape/qshape.pl -s active| head"
        >>
        >> I have a reasonably "normal" number of deferred emails (no more than 100
        >> messages).
        >>
        >> Nevertheless, I'm doing recipient validation for each mailbox that I
        >> manage and verification on each email of every domain for which I
        >> forward messages.
        >>
        >> I fear that the problem is that for each session I can have an unsettled
        >> number of messages sent over that session (It could be happen? If yes,
        >> It could be depend on MTA settings?) other then an unsettled size of
        >> SMTP traffic (which it determs the latency of messages and it could make
        >> congestion of postfix active queue more or less heavy).

        Could someone give me some hint about this issue?

        TIA,

        rocsca
      • Victor Duchovni
        ... I have no idea what this issue is, and I doubt anyone else does either. Unless you can present concrete information, rather than vague guesses, it is
        Message 3 of 10 , Feb 3, 2009
        • 0 Attachment
          On Tue, Feb 03, 2009 at 09:10:50PM +0100, Rocco Scappatura wrote:

          > >> I fear that the problem is that for each session I can have an unsettled
          > >> number of messages sent over that session (It could be happen? If yes,
          > >> It could be depend on MTA settings?) other then an unsettled size of
          > >> SMTP traffic (which it determs the latency of messages and it could make
          > >> congestion of postfix active queue more or less heavy).
          >
          > Could someone give me some hint about this issue?

          I have no idea what "this issue" is, and I doubt anyone else does either.
          Unless you can present concrete information, rather than vague guesses,
          it is unlikely that you will get much help.

          Postfix is an I/O bandwidth limited MTA, running within fixed concurrency
          limits. When you add content filters, the filters may become CPU-limited.

          Throughput = Concurrency / Latency.

          If you are seeing low throughput, but the system has enough resources
          to provide more throughput, your concurrency may be too low, or your
          delivery agents are all tied up timing out deliveries to dead destinations
          (abnormally high latency).

          If you have run out of CPU, I/O or network bandwidth, add more hardware,
          or reduce demand for that resource.

          Sadly, you have to find the reason you are experiencing congestion, and
          quantify this with relevant measurements.

          --
          Viktor.

          Disclaimer: off-list followups get on-list replies or get ignored.
          Please do not ignore the "Reply-To" header.

          To unsubscribe from the postfix-users list, visit
          http://www.postfix.org/lists.html or click the link below:
          <mailto:majordomo@...?body=unsubscribe%20postfix-users>

          If my response solves your problem, the best way to thank me is to not
          send an "it worked, thanks" follow-up. If you must respond, please put
          "It worked, thanks" in the "Subject" so I can delete these quickly.
        • Rocco Scappatura
          Victor, ... I agree with all of your argumntation. But, basically my question is another one. Maybe I m wrong to try to submit the problem describing whatever
          Message 4 of 10 , Feb 3, 2009
          • 0 Attachment
            Victor,

            >> >> I fear that the problem is that for each session I can have an
            >> unsettled
            >> >> number of messages sent over that session (It could be happen? If
            >> yes,
            >> >> It could be depend on MTA settings?) other then an unsettled size of
            >> >> SMTP traffic (which it determs the latency of messages and it could
            >> make
            >> >> congestion of postfix active queue more or less heavy).
            >>
            >> Could someone give me some hint about this issue?
            >
            > I have no idea what "this issue" is, and I doubt anyone else does either.
            > Unless you can present concrete information, rather than vague guesses,
            > it is unlikely that you will get much help.
            >
            > Postfix is an I/O bandwidth limited MTA, running within fixed concurrency
            > limits. When you add content filters, the filters may become CPU-limited.
            >
            > Throughput = Concurrency / Latency.
            >
            > If you are seeing low throughput, but the system has enough resources
            > to provide more throughput, your concurrency may be too low, or your
            > delivery agents are all tied up timing out deliveries to dead destinations
            > (abnormally high latency).
            >
            > If you have run out of CPU, I/O or network bandwidth, add more hardware,
            > or reduce demand for that resource.
            >
            > Sadly, you have to find the reason you are experiencing congestion, and
            > quantify this with relevant measurements.

            I agree with all of your argumntation. But, basically my question is
            another one. Maybe I'm wrong to try to submit the problem describing
            whatever comes around which only contributes to complicate the
            understending for the list. I'm sorry for this.

            Returning to my question, I'm trying to understand:

            1) Once a client (or another MTA) establish a TCP connection with
            listening port bounded by the SMTP daemon of Postfix, could happen that
            more then one email messages are sent over that TCP connection, before it
            is closed?

            2) If 1), is there any limit on the number of messages that could be sent
            over that TCP connection?

            3) Could the receiving MTA (i.e.: Postfix) decide how much times a TCP
            connection could used to transmit a messages by a client?

            I'm sorry again if my answer are trivial or that make no sense..

            rocsca
          • Victor Duchovni
            ... Sure this is possible, but it is unlikel to significantly impact your queues. ... No. ... Enforcing such limits is unwise. The solution causes more harm
            Message 5 of 10 , Feb 3, 2009
            • 0 Attachment
              On Tue, Feb 03, 2009 at 11:59:37PM +0100, Rocco Scappatura wrote:

              > Returning to my question, I'm trying to understand:
              >
              > 1) Once a client (or another MTA) establish a TCP connection with
              > listening port bounded by the SMTP daemon of Postfix, could happen that
              > more then one email messages are sent over that TCP connection, before it
              > is closed?

              Sure this is possible, but it is unlikel to significantly impact your
              queues.

              > 2) If 1), is there any limit on the number of messages that could be sent
              > over that TCP connection?

              No.

              > 3) Could the receiving MTA (i.e.: Postfix) decide how much times a TCP
              > connection could used to transmit a messages by a client?

              Enforcing such limits is unwise. The solution causes more harm than the
              perceived problem.

              There is no evidence that sender-side connection re-use has any material
              impact on your queues. If you do want to enforce such limits, they should
              be applied selectively to just IP sources with poor "reputations".

              --
              Viktor.

              Disclaimer: off-list followups get on-list replies or get ignored.
              Please do not ignore the "Reply-To" header.

              To unsubscribe from the postfix-users list, visit
              http://www.postfix.org/lists.html or click the link below:
              <mailto:majordomo@...?body=unsubscribe%20postfix-users>

              If my response solves your problem, the best way to thank me is to not
              send an "it worked, thanks" follow-up. If you must respond, please put
              "It worked, thanks" in the "Subject" so I can delete these quickly.
            • Rocco Scappatura
              Thanks Viktor, ... Indeed, it would be nice to have a tool that assigns a poor reputation to an IP source that impact on the queues.. Maybe one of these tool
              Message 6 of 10 , Feb 3, 2009
              • 0 Attachment
                Thanks Viktor,

                >> 1) Once a client (or another MTA) establish a TCP connection with
                >> listening port bounded by the SMTP daemon of Postfix, could happen that
                >> more then one email messages are sent over that TCP connection, before
                >> it
                >> is closed?
                >
                > Sure this is possible, but it is unlikel to significantly impact your
                > queues.
                >
                >> 2) If 1), is there any limit on the number of messages that could be
                >> sent
                >> over that TCP connection?
                >
                > No.
                >
                >> 3) Could the receiving MTA (i.e.: Postfix) decide how much times a TCP
                >> connection could used to transmit a messages by a client?
                >
                > Enforcing such limits is unwise. The solution causes more harm than the
                > perceived problem.
                >
                > There is no evidence that sender-side connection re-use has any material
                > impact on your queues. If you do want to enforce such limits, they should
                > be applied selectively to just IP sources with poor "reputations".
                >

                Indeed, it would be nice to have a tool that assigns a poor reputation to
                an IP source that impact on the queues.. Maybe one of these tool could be
                a Policyd server? Simply imposing a quota on the number of messages that
                could be sent in a unit of time? Or there exists some of more refined, for
                what you know?

                rocsca
              • Victor Duchovni
                ... No, it would be nice to have to tools that assign fewer resources to senders with poor reputations, but just sending you a lot of legitimate mail is not
                Message 7 of 10 , Feb 3, 2009
                • 0 Attachment
                  On Wed, Feb 04, 2009 at 02:21:31AM +0100, Rocco Scappatura wrote:

                  > > There is no evidence that sender-side connection re-use has any material
                  > > impact on your queues. If you do want to enforce such limits, they should
                  > > be applied selectively to just IP sources with poor "reputations".
                  >
                  > Indeed, it would be nice to have a tool that assigns a poor reputation to
                  > an IP source that impact on the queues.. Maybe one of these tool could be
                  > a Policyd server? Simply imposing a quota on the number of messages that
                  > could be sent in a unit of time? Or there exists some of more refined, for
                  > what you know?

                  No, it would be nice to have to tools that assign fewer resources to
                  senders with poor reputations, but just sending you a lot of legitimate
                  mail is not sufficient cause.

                  I still don't see why you believe that connection re-use by high-volume
                  senders is the cause of the imbalance you observe.

                  --
                  Viktor.

                  Disclaimer: off-list followups get on-list replies or get ignored.
                  Please do not ignore the "Reply-To" header.

                  To unsubscribe from the postfix-users list, visit
                  http://www.postfix.org/lists.html or click the link below:
                  <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                  If my response solves your problem, the best way to thank me is to not
                  send an "it worked, thanks" follow-up. If you must respond, please put
                  "It worked, thanks" in the "Subject" so I can delete these quickly.
                Your message has been successfully submitted and would be delivered to recipients shortly.