Loading ...
Sorry, an error occurred while loading the content.

Re: mail lost

Expand Messages
  • pobox@verysmall.org
    ... That s true. That s why I asked for help here. I am not very good, but if it is reproducible, I read and try and eventually find out. It was reproducible
    Message 1 of 13 , Apr 1, 2007
    • 0 Attachment
      mouss wrote:
      > pobox@... wrote:
      >> mouss wrote:
      >>> pobox@... wrote:
      >>>> Michael Wang wrote:
      >>>>> pobox@... wrote:
      >>>>>> hello,
      >>>>>>
      >>>>>> yesterday all mail sent from my workstation (tried Thunderbird and
      >>>>>> Outlook Express) seemed sent as usual, but did not arrive at the
      >>>>>> recipient. mail sent from Squirrelmail and another application on
      >>>>>> the server - did arrive.
      >>>>>>
      >>>>>> the mail that did not arrive could not be found in any queue.
      >>>>>>
      >>>>>> today everything work, as it has been working for the past months
      >>>>>> (no changes have been done to the mail server in the past days).
      >>>>>>
      >>>>>> could somebody give me a hint what it might be.
      >>>>>>
      >>>>>> thanks,
      >>>>>> iv
      >>>>>
      >>>>> What did the logs say?
      >>>>
      >>>> The mail activity goes into /var/log/maillog, but I could not find
      >>>> any entries about the lost mail there.
      >>>>
      >>>
      >>> then the MUAs used another MTA or they queued the messages for later.
      >>> postfix logs all transactions.
      >>
      >> Could you give me a tip how to debug this process, where are the
      >> points to check/monitor, except the maillog and queue (the mail server
      >> is Postfix and Cyrus)?
      >>
      >> The account is IMAP, do not know if this helps.
      >>
      >> What is 'sure' is that -
      >>
      >> - the MUA setting (SMTP domain) hasn't been changed
      >> - the SMTP domain itself hasn't been changed
      >> - the mail server configuration hasn't been changed
      >> - the problem seemed to appear and disappear on its own (nothing was
      >> done to fix it - one day it did not work, the next it worked)
      >> - the MUA (both Thunderbird and Outlook Express) reported 'mail sent'
      >> - the lost mail could not be found in any queue
      >
      > unfortunately, if you can't reproduce the problem, it'll be hard to find
      > out...

      That's true. That's why I asked for help here. I am not very good, but
      if it is reproducible, I read and try and eventually find out. It was
      reproducible on that day, but I did not advance enough with the
      debugging then.

      > if you are "lucky", the mail was delivered to a mailbox instead of to
      > cyrus. look in /var/mail/ (or whatever is the default delivery directory
      > for your MTA).

      No, it was intended to recipients outside of the mail server.

      > other possibilities:
      > - at some moment, DNS returned another IP as the MTA. the MUA then sent
      > to that MTA

      Yes, somebody else mentioned the same possibility. But how is that
      possible, it sounds scary...

      > - some firewall rule redirect smtp packets to another MTA

      Hm... can this happen 'out of the blue' and fix itself...

      Hm... mystery.

      Anyway, thank you for your time and possible reasons. Will help me to
      advance more next time.

      Iv
    • pobox@verysmall.org
      ... To my knowledge nothing has been changed on the server (it s a dedicated server) on that day (the problem as about half a day or so). Before we had qmail,
      Message 2 of 13 , Apr 1, 2007
      • 0 Attachment
        Sean Ryan wrote:
        >> Could you give me a tip how to debug this process, where are the points
        >> to check/monitor, except the maillog and queue (the mail server is
        >> Postfix and Cyrus)?
        >>
        >> The account is IMAP, do not know if this helps.
        >>
        >> What is 'sure' is that -
        >>
        >> - the MUA setting (SMTP domain) hasn't been changed
        >> - the SMTP domain itself hasn't been changed
        >> - the mail server configuration hasn't been changed
        >> - the problem seemed to appear and disappear on its own (nothing was
        >> done to fix it - one day it did not work, the next it worked)
        >> - the MUA (both Thunderbird and Outlook Express) reported 'mail sent'
        >> - the lost mail could not be found in any queue
        >>
        >> (Unless something has been hacked, which is excluded as a possibility
        >> for now.)
        >
        > Did anything happen like rebooting ( MUA machine or SMTP machine )? It
        > is possible that this might cause something that was working to stop (
        > or vice versa ). A good example is that if another mta ( such as exim
        > which logs to a different location ) started up instead of postfix due
        > to a configuration error.

        To my knowledge nothing has been changed on the server (it's a dedicated
        server) on that day (the problem as about half a day or so). Before we
        had qmail, which is not un-installed, as far as I know. But I am not
        aware of any event (i.e. reboot of something) which might have caused
        configuration change or some of the possibilities described by you.

        > One thing that I'm not clear on from your description ( forgive me if I
        > just missed it ), was the exact situation. You have a workstation, and
        > this workstation is NOT running an SMTP server ( ie just a windows
        > workstation ) ?

        Yes.

        There is a FreeBSD dedicated server with Postfix/Cyrus, Squirrelmail and
        another php application - the Squirrelmail and the php application did
        not have any problems to send mail. And there is a Windows workstation
        where I tried Thunderbird (my primary mail client) and Outlook Express
        (which I set for the test) - during this period both gave the impression
        that mail is sent, but mail did not arrive, neither was logged in
        maillog, nor was it in the queue, to my experience (I might have missed
        something) I could not find any trace of the mail with 'grep'. Somebody
        advised me to try to send mail using telnet from the Workstation, but at
        this moment I did not know how and when I found out how on the next day,
        the problem did not exist anymore. So I can't tell if telnet would have
        worked (resp. I do not know what telnet would have reported, i.e.
        somebody mentioned that it might be a DNS problem).

        > 1) Make sure 100% that the connection you think
        > is being made is. If you have access to the smtp server, then do a
        > tcpdump on the server side and watch yourself connect.

        Oh, thanks a lot! This is actually my main question - tools to debug and
        locations to look. I guess now I can't use it, but I wrote it down in my
        debug notes for when it happens next time.

        > Something like
        > this:
        >
        > There are many options, and if you are inclined you can actually capture
        > the full data as it comes in ( thus being able to verify 100% ). You
        > could also install ethereal/wireshark ... which does the exact same
        > thing ... but in a nice gui form factor.

        Yes! I have wireshark, but it did not occur to me that I can use it to
        see what the MUAs (Thunderbird/Outlook Express) actually do. Put it also
        in my debug notes for next time, thanks.

        > A) On the Client Side ( ie where outlook is ). Also note, if this is a
        > window machine, you may not see what you type in because echo might be
        > turned off :
        > telnet smtp.server.com 25

        Yes, as I already learned how to use telnet in order to test the SMTP.
        Unfortunately after the problem was gone :/

        > B) On the Server Side ( ie where postfix is running NOTE: I use port 80
        > here in the tcpdump, this would be replaced with 25 ):
        > root@server:~# tcpdump -s0 -XXX port 80
        > tcpdump: verbose output suppressed, use -v or -vv for full protocol
        > decode
        > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
        > 19:00:03.514386 IP client.ip.addr.45482 > server.www: S
        > 268794353:268794353(0) win 5840 <mss 1452,sackOK,timestamp 158639826
        > 0,nop,wscale 7>
        > 0x0000: 0011 09d1 95a0 0001 e8d5 b309 0800 4500
        > ..............E.
        > 0x0010: 003c df23 4000 2d06 dcca d863 6188 40fb
        > .<.#@.-....ca.@.
        > 0x0020: 16e7 b1aa 0050 1005 79f1 0000 0000 a002
        > .....P..y.......
        > 0x0030: 16d0 b332 0000 0204 05ac 0402 080a 0974
        > ...2...........t
        > 0x0040: a6d2 0000 0000 0103 0307 ..........
        > 19:00:03.516255 IP server.www > client.ip.addr.45482: S
        > 1611562685:1611562685(0) ack 268794354 win 5792 <mss
        > 1460,sackOK,timestamp 1143357142 158639826,nop,wscale 2>

        Really thanks a lot. This is what I wanted to know - tools to debug.

        > The data is mostly unimportant, what you are looking for is that the
        > connection came in.

        Yes, correct.

        > *IF* that is the case, then you might need to dig
        > deeper to ensure that the full message is transferred. Also make sure
        > that you see the same connection happening while using outlook.

        Yes! This is a great beginning. I assume this will give a lot of data
        (or no data, which is also something) to help solve the problem, or move on.

        > 2) Conduct a command line smtp session ... make sure everything looks
        > ok ... something like:
        > sryan@oberon:~$ telnet inbound.evercrack.com.emailmx.com 25
        > Trying 64.97.156.1...
        > Connected to inbound.evercrack.com.emailmx.com.
        > Escape character is '^]'.
        > 220 Email Defense System ESMTP EMD
        > ehlo deimos.evercrack.com
        > 250-emd2-imf4.emaildefenseservice.com
        > 250-PIPELINING
        > 250-SIZE 25000000
        > 250-ETRN
        > 250-ENHANCEDSTATUSCODES
        > 250 8BITMIME
        > mail from: <sryan@...>
        > 250 2.1.0 Ok
        > rcpt to: <sryan@...>
        > 250 2.1.5 Ok
        > data
        > 354 End data with <CR><LF>.<CR><LF>
        > Subject: testing
        > .
        > 250 OK

        Yes.

        > 3) If this is not good enough ... do something client related where you
        > check the outbound smtp connections. It's totally anti-rfc, but perhaps
        > outlook is decided to deliver the mail right to the MX, bypassing your
        > outbound settings ( this is a long shot, but you can check anyway as a
        > last resort ).

        OK

        > 4) Check the mail server to ensure that the process you think is
        > running, in fact is.

        Yes, I wanted to do this, but I could not figure out what is the smtp
        process.

        Thanks for this hint as well.

        > A) On the server side, see who is listening on port 25
        > #netstat -tanwup | grep LISTEN | grep 25
        > tcp 0 0 127.0.0.1:25 0.0.0.0:*
        > LISTEN 3123/master
        >
        > B) Then check what he's got open:
        > # lsof -i | grep 3123
        > master 3123 root 11u IPv4 8960 TCP localhost:smtp
        > (LISTEN)
        >
        > C) And check the proc info:
        > root@oberon:~/code/rotk/lib/OGRE# ls -al /proc/3123
        > total 0
        > dr-xr-xr-x 5 root root 0 2007-03-30 07:30 .
        > dr-xr-xr-x 122 root root 0 2007-03-24 11:39 ..
        > dr-xr-xr-x 2 root root 0 2007-03-31 20:19 attr
        > -r-------- 1 root root 0 2007-03-31 20:19 auxv
        > -r--r--r-- 1 root root 0 2007-03-30 23:32 cmdline
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 cpuset
        > lrwxrwxrwx 1 root root 0 2007-03-31 20:17 cwd -> /var/spool/postfix
        > -r-------- 1 root root 0 2007-03-31 20:19 environ
        > lrwxrwxrwx 1 root root 0 2007-03-31 07:30 exe ->
        > /usr/lib/postfix/master
        > dr-x------ 2 root root 0 2007-03-31 20:05 fd
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 maps
        > -rw------- 1 root root 0 2007-03-31 20:19 mem
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 mounts
        > -r-------- 1 root root 0 2007-03-31 20:19 mountstats
        > -rw-r--r-- 1 root root 0 2007-03-31 20:19 oom_adj
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 oom_score
        > lrwxrwxrwx 1 root root 0 2007-03-31 20:17 root -> /
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 smaps
        > -r--r--r-- 1 root root 0 2007-03-30 23:32 stat
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 statm
        > -r--r--r-- 1 root root 0 2007-03-30 23:32 status
        > dr-xr-xr-x 3 root root 0 2007-03-31 20:19 task
        > -r--r--r-- 1 root root 0 2007-03-31 20:19 wchan

        These are precious tips, thank you so much.

        > If I go deeper than that ... we're getting into advanced
        > troubleshooting. In all honesty, try the simple stuff first, this is
        > almost 99% chance that this is somehow user error in some way. The
        > bottom line is that postfix logs all transactions ... so if the mail was
        > "sent", and postfix has no record whatsoever ... then either you're
        > checking the wrong postfix, or your client is sending to the wrong
        > place.

        Yes. I am sure that if I knew all this then, I would have found out enough.

        Thank you so much for your time to type all this.

        If I catch it, I'll write back.

        Iv

        PS I replied back to the list... hope it's OK.
      • mouss
        ... In fact, a more probable hole is an anti-virus silently quarantining mail. if you are running an AV, look in its quarantine. another possibility is a
        Message 3 of 13 , Apr 1, 2007
        • 0 Attachment
          pobox@... wrote:
          >
          >> if you are "lucky", the mail was delivered to a mailbox instead of to
          >> cyrus. look in /var/mail/ (or whatever is the default delivery
          >> directory for your MTA).
          >
          > No, it was intended to recipients outside of the mail server.
          >

          In fact, a more probable "hole" is an anti-virus silently quarantining
          mail. if you are running an AV, look in its quarantine.

          another possibility is a honeypot triggerred by an IDS system.
        • mouss
          ... try resending the _same_ message from the same machine. if the AV quarantines it, then you get the guilty. now, the AV config may have changed (or it may
          Message 4 of 13 , Apr 1, 2007
          • 0 Attachment
            pobox@... wrote:
            >
            > There is a FreeBSD dedicated server with Postfix/Cyrus, Squirrelmail
            > and another php application - the Squirrelmail and the php application
            > did not have any problems to send mail. And there is a Windows
            > workstation where I tried Thunderbird (my primary mail client) and
            > Outlook Express (which I set for the test) - during this period both
            > gave the impression that mail is sent, but mail did not arrive,
            > neither was logged in maillog, nor was it in the queue, to my
            > experience (I might have missed something) I could not find any trace
            > of the mail with 'grep'. Somebody advised me to try to send mail using
            > telnet from the Workstation, but at this moment I did not know how and
            > when I found out how on the next day, the problem did not exist
            > anymore. So I can't tell if telnet would have worked (resp. I do not
            > know what telnet would have reported, i.e. somebody mentioned that it
            > might be a DNS problem).

            try resending the _same_ message from the same machine. if the AV
            quarantines it, then you get the guilty. now, the AV config may have
            changed (or it may have dynamic rules....) so you are not guaranteed to
            reproduce the problem, but it's worth to try. and as I said in my
            previous post, check the AV quarantine. of course, this all assumes you
            are (were) running an AV on the client box.
          • pobox@verysmall.org
            ... Oh, thanks. This is a good hint. I check the guaranteed messages and there weren t such. But added it to my debug list for next time it happens. ... What
            Message 5 of 13 , Apr 2, 2007
            • 0 Attachment
              mouss wrote:
              > pobox@... wrote:
              >>
              >>> if you are "lucky", the mail was delivered to a mailbox instead of to
              >>> cyrus. look in /var/mail/ (or whatever is the default delivery
              >>> directory for your MTA).
              >>
              >> No, it was intended to recipients outside of the mail server.
              >
              > In fact, a more probable "hole" is an anti-virus silently quarantining
              > mail. if you are running an AV, look in its quarantine.

              Oh, thanks. This is a good hint. I check the guaranteed messages and
              there weren't such. But added it to my debug list for next time it happens.

              > another possibility is a honeypot triggerred by an IDS system.

              What is this?
            • mouss
              ... a honeypot is a trap. an IDS is an Intrusion Detection System. some IDS software will listen on known ports and acts like a server, except that it doesn t
              Message 6 of 13 , Apr 2, 2007
              • 0 Attachment
                pobox@... wrote:
                > mouss wrote:
                > > pobox@... wrote:
                > >>
                > >>> if you are "lucky", the mail was delivered to a mailbox instead of to
                > >>> cyrus. look in /var/mail/ (or whatever is the default delivery
                > >>> directory for your MTA).
                > >>
                > >> No, it was intended to recipients outside of the mail server.
                > >
                > > In fact, a more probable "hole" is an anti-virus silently quarantining
                > > mail. if you are running an AV, look in its quarantine.
                >
                > Oh, thanks. This is a good hint. I check the guaranteed messages and
                > there weren't such. But added it to my debug list for next time it
                > happens.
                >
                > > another possibility is a honeypot triggerred by an IDS system.
                >
                > What is this?
                >
                >

                a honeypot is a trap. an IDS is an Intrusion Detection System. some IDS
                software will listen on known ports and acts like a server, except that
                it doesn't really do what the attacker expects.

                This is not very probable, but you never know.
              • pobox@verysmall.org
                ... OK Got it. Thanks a lot for this and for all your previous help. Iv
                Message 7 of 13 , Apr 5, 2007
                • 0 Attachment
                  mouss wrote:
                  > a honeypot is a trap. an IDS is an Intrusion Detection System. some IDS
                  > software will listen on known ports and acts like a server, except that
                  > it doesn't really do what the attacker expects.
                  >
                  > This is not very probable, but you never know.

                  OK Got it.

                  Thanks a lot for this and for all your previous help.

                  Iv
                Your message has been successfully submitted and would be delivered to recipients shortly.