Loading ...
Sorry, an error occurred while loading the content.

Re: Problem sending a DOS text file from Linux to Windows.

Expand Messages
  • scott.postfix@scottrix.co.uk
    ... From what I have managed to find on the web the SMTP protocol uses CRLF text to transfer text data around, now I assume that postfix is doing the
    Message 1 of 19 , Feb 1, 2008
    • 0 Attachment
      On Fri, Feb 01, 2008 at 09:56:50AM -0500, Wietse Venema wrote:
      >scott.postfix@...:
      >> Hi
      >>
      >> I am having a problem sending a Dos (CRLF) text file from Linux to Windows
      >> as an attachment (mimetype = text/plain, encoding = 7bit). I am using mutt
      >> as my email client and postfix (2.4.5) as my MTA. When the file is
      >> received the CRLF has been replaced by CRLFCRLF, if I send the same file to
      >> myself (on Linux) then the CRLF line endings have been replaced with LFLF.
      >> Creating a double line spacing effect.
      >>
      >> To try to narrow down the problem I replaced the MTA that mutt uses with a
      >> bash script to dump the file out to a log. This log showed that the
      >> message still contained just one CRLF line ending on each line.
      >>
      >> I then ran tcpdump so monitor the outgoing smtp connect that postfix was
      >> making. This showed that the line ending had changed from CRLF to CRCRLF.
      >> I am guessing that something somewhere is changing the LF at the end of the
      >> lines to CRLF despite the fact that each line already contains this
      >> combination.
      >>
      >> I am not sure where to go from here to find out what is going on, any help
      >> would be appreciated. If you require anymore information please let me
      >> know.
      >
      >UNIX systems don't use DOS textfile formats. Use the unix2dos
      >command (or similar tool) to properly handle DOS text files on a
      >UNIX system.

      From what I have managed to find on the web the SMTP protocol uses CRLF
      text to transfer text data around, now I assume that postfix is doing the
      conversion from unix text (LF) to SMTP text (CRLF), if the text is already
      in that format and since it is text data and not binary is there a problem
      with leaving text that is already (CRLF) alone ?

      Regards,

      Scott.
    • Wietse Venema
      ... No this is incorrect. Local text MUST be in UNIX text format. Postfix will convert between network formats and local formats. To send DOS a textfile from a
      Message 2 of 19 , Feb 1, 2008
      • 0 Attachment
        scott.postfix@...:
        > >UNIX systems don't use DOS textfile formats. Use the unix2dos
        > >command (or similar tool) to properly handle DOS text files on a
        > >UNIX system.
        >
        > From what I have managed to find on the web the SMTP protocol uses CRLF
        > text to transfer text data around, now I assume that postfix is doing the
        > conversion from unix text (LF) to SMTP text (CRLF), if the text is already
        > in that format and since it is text data and not binary is there a problem
        > with leaving text that is already (CRLF) alone ?

        No this is incorrect.

        Local text MUST be in UNIX text format. Postfix will convert
        between network formats and local formats.

        To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
        tool or send the file as a base64 encoded attachment.

        Wietse
      • scott.postfix@scottrix.co.uk
        ... Are you saying that a DOS text file on Linux is actually a binary file and should not have a text/plain mime type ? If I wanted to modify this behaviour
        Message 3 of 19 , Feb 1, 2008
        • 0 Attachment
          >Local text MUST be in UNIX text format. Postfix will convert between
          >network formats and local formats.

          Are you saying that a DOS text file on Linux is actually a "binary" file
          and should not have a text/plain mime type ?

          If I wanted to modify this behaviour and "break" my system where in the
          postfix code does it do this conversion ?

          Scott
        • Victor Duchovni
          ... tr -d 015 real_text_file Then mail the real_text_file . If you want to undertake independent Postfix development, ... good luck. --
          Message 4 of 19 , Feb 1, 2008
          • 0 Attachment
            On Fri, Feb 01, 2008 at 03:46:05PM +0000, scott.postfix@... wrote:

            > >Local text MUST be in UNIX text format. Postfix will convert between
            > >network formats and local formats.
            >
            > Are you saying that a DOS text file on Linux is actually a "binary" file
            > and should not have a text/plain mime type ?
            >
            > If I wanted to modify this behaviour and "break" my system where in the
            > postfix code does it do this conversion ?

            tr -d '\015' < dos_text_file > real_text_file

            Then mail the "real_text_file". If you want to undertake independent
            Postfix development, ... good luck.

            --
            Viktor.

            Disclaimer: off-list followups get on-list replies or get ignored.
            Please do not ignore the "Reply-To" header.

            To unsubscribe from the postfix-users list, visit
            http://www.postfix.org/lists.html or click the link below:
            <mailto:majordomo@...?body=unsubscribe%20postfix-users>

            If my response solves your problem, the best way to thank me is to not
            send an "it worked, thanks" follow-up. If you must respond, please put
            "It worked, thanks" in the "Subject" so I can delete these quickly.
          • Erwan David
            Le Fri 1/02/2008, scott.postfix@scottrix.co.uk disait ... If you want the file to keep the exact binary representation through transport, then the transport
            Message 5 of 19 , Feb 1, 2008
            • 0 Attachment
              Le Fri 1/02/2008, scott.postfix@... disait
              >> Local text MUST be in UNIX text format. Postfix will convert between
              >> network formats and local formats.
              >
              > Are you saying that a DOS text file on Linux is actually a "binary" file
              > and should not have a text/plain mime type ?

              If you want the file to keep the exact binary representation through
              transport, then the transport must be told to consider it binary.

              --
              Erwan
            • Wietse Venema
              ... Different operating systems represent end-of-line conventions in different ways. 1) UNIX stores the end-of-line as a LF character. 2) DOS stores the
              Message 6 of 19 , Feb 1, 2008
              • 0 Attachment
                scott.postfix@...:
                > >Local text MUST be in UNIX text format. Postfix will convert between
                > >network formats and local formats.
                >
                > Are you saying that a DOS text file on Linux is actually a "binary" file
                > and should not have a text/plain mime type ?

                Different operating systems represent end-of-line conventions in
                different ways.

                1) UNIX stores the end-of-line as a LF character.

                2) DOS stores the end-of-line as a CRLF pair.

                3) Other systems don't store the end-of-line at all, instead
                they store a record length, or worse, they assume all records
                have the same fixed length, and the length is a global attribute
                for the entire file.

                If you want to send the TEXT in a DOS file from a UNIX box, convert
                the file to UNIX format before passing it to the mail system.

                If you want to send a DOS textfile in DOS textfile format,
                use binary MIME type and base64 encoding.

                Wietse
              • Richard Foley
                ... You re not listening to the answer: ... -- Richard Foley Ciao - shorter than aufwiedersehen http://www.rfi.net/
                Message 7 of 19 , Feb 1, 2008
                • 0 Attachment
                  On Friday 01 February 2008 16:46, scott.postfix@... wrote:
                  > >Local text MUST be in UNIX text format. Postfix will convert between
                  > >network formats and local formats.
                  >
                  > Are you saying that a DOS text file on Linux is actually a "binary" file
                  > and should not have a text/plain mime type ?
                  >
                  > If I wanted to modify this behaviour and "break" my system where in the
                  > postfix code does it do this conversion ?
                  >
                  You're not listening to the answer:

                  Wietse said:

                  > To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                  > tool or send the file as a base64 encoded attachment.
                  >

                  --
                  Richard Foley
                  Ciao - shorter than aufwiedersehen

                  http://www.rfi.net/
                • scott.postfix@scottrix.co.uk
                  ... Let me try to put it another way. How do I know when I need to convert the file before I send it ? I don t intend to manually check every single text file
                  Message 8 of 19 , Feb 4, 2008
                  • 0 Attachment
                    On Fri, Feb 01, 2008 at 05:30:04PM +0100, Richard Foley wrote:
                    >You're not listening to the answer:
                    >
                    >Wietse said:
                    >
                    >> To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                    >> tool or send the file as a base64 encoded attachment.

                    Let me try to put it another way. How do I know when I need to convert the
                    file before I send it ? I don't intend to manually check every single text
                    file that I send to see if I should then manually convert it first.

                    Using the UNIX file utility with the "-i" option which lists the mime type
                    it gives:

                    text/plain; charset=us-ascii

                    If this is correct, then shouldn't the MTA handle this mime type without
                    screwing up the file ?

                    Programmatically it is not difficult, so I assume that the negativity that
                    I am getting is because I am not getting something, so please please
                    explain why changing the code to cope with this would be a bad idea.

                    Scott.

                    PS. Tried the same thing with exim and this works as it should, i.e.
                    doesn't introduce the extra CR in the attachment. I don't want to switch
                    MTAs as I believe Postfix is better in many ways, but I also don't want to
                    send out broken emails.
                  • Alexey Lobanov
                    Hello. ... Possibly nohow, as fromdos is a safe filter. It does nothing bad with an unix text. $ man fromdos The default is to remove carriage returns only
                    Message 9 of 19 , Feb 4, 2008
                    • 0 Attachment
                      Hello.

                      04.02.2008 12:00, scott.postfix@... пишет:

                      > On Fri, Feb 01, 2008 at 05:30:04PM +0100, Richard Foley wrote:
                      >> You're not listening to the answer:
                      >>
                      >> Wietse said:
                      >>
                      >>> To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                      >>> tool or send the file as a base64 encoded attachment.
                      >
                      > Let me try to put it another way. How do I know when I need to convert
                      > the file before I send it ?

                      Possibly nohow, as "fromdos" is a safe filter. It does nothing bad with
                      an unix text.

                      $ man fromdos
                      "The default is to remove carriage returns only if they are followed
                      by line feeds."

                      Alexey
                    • Victor Duchovni
                      ... If you are sending data files (binary FTP over email), you don t convert at all, just base64 encode the file bits and send. If you are sending text files
                      Message 10 of 19 , Feb 4, 2008
                      • 0 Attachment
                        On Mon, Feb 04, 2008 at 09:00:34AM +0000, scott.postfix@... wrote:

                        > >>To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                        > >>tool or send the file as a base64 encoded attachment.
                        >
                        > Let me try to put it another way. How do I know when I need to convert the
                        > file before I send it ? I don't intend to manually check every single text
                        > file that I send to see if I should then manually convert it first.

                        If you are sending data files (binary FTP over email), you don't convert
                        at all, just base64 encode the file bits and send. If you are sending text
                        files (ASCII FTP over email), and your text files are not actually UNIX
                        text files, you need to convert them to unix text files before sending.

                        > If this is correct, then shouldn't the MTA handle this mime type without
                        > screwing up the file ?

                        MTAs transmit messages, MIME encoding is the responsibility of an *MUA*.
                        Postfix is not a MUA.

                        --
                        Viktor.

                        Disclaimer: off-list followups get on-list replies or get ignored.
                        Please do not ignore the "Reply-To" header.

                        To unsubscribe from the postfix-users list, visit
                        http://www.postfix.org/lists.html or click the link below:
                        <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                        If my response solves your problem, the best way to thank me is to not
                        send an "it worked, thanks" follow-up. If you must respond, please put
                        "It worked, thanks" in the "Subject" so I can delete these quickly.
                      • mouss
                        ... it s up to you (to switch to exim or not), but in principle you can t do # sendmail ...
                        Message 11 of 19 , Feb 4, 2008
                        • 0 Attachment
                          scott.postfix@... wrote:
                          > On Fri, Feb 01, 2008 at 05:30:04PM +0100, Richard Foley wrote:
                          >> You're not listening to the answer:
                          >>
                          >> Wietse said:
                          >>
                          >>> To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                          >>> tool or send the file as a base64 encoded attachment.
                          >
                          > Let me try to put it another way. How do I know when I need to convert
                          > the file before I send it ? I don't intend to manually check every
                          > single text file that I send to see if I should then manually convert
                          > it first.
                          >
                          > Using the UNIX file utility with the "-i" option which lists the mime
                          > type it gives:
                          >
                          > text/plain; charset=us-ascii
                          >
                          > If this is correct, then shouldn't the MTA handle this mime type
                          > without screwing up the file ?
                          >
                          > Programmatically it is not difficult, so I assume that the negativity
                          > that I am getting is because I am not getting something, so please
                          > please explain why changing the code to cope with this would be a bad
                          > idea.
                          >
                          > Scott.
                          >
                          > PS. Tried the same thing with exim and this works as it should, i.e.
                          > doesn't introduce the extra CR in the attachment. I don't want to
                          > switch MTAs as I believe Postfix is better in many ways, but I also
                          > don't want to send out broken emails.

                          it's up to you (to switch to exim or not), but in principle you can't do

                          # sendmail ... < foo.mp4

                          mail data must obey some constraints.

                          with text files, you can still do
                          # dos2unix foo.txt | sendmail ....

                          the dos2unix command will convert the file to "unix format" if it was in
                          dos format.
                        • scott.postfix@scottrix.co.uk
                          Sorry for being a pain, but I want to get this clear. I can only see two things that could be wrong to get to the problem I am having. 1) Postfix is broken
                          Message 12 of 19 , Feb 4, 2008
                          • 0 Attachment
                            Sorry for being a pain, but I want to get this clear. I can only see two
                            things that could be wrong to get to the problem I am having.

                            1) Postfix is broken and should be detecting when text/plain attachments
                            have CRLF and not add the CR.

                            2) My MUA (mutt) is broken and is incorrectly identifying a DOS text file
                            as text/plain when it is not.

                            From what is being said here 1) is not the case, so is the problem 2) or is
                            there a third option I haven't thought of.

                            Thanks for your patience,

                            Scott.


                            On Mon, Feb 04, 2008 at 09:17:54AM -0500, Victor Duchovni wrote:
                            >On Mon, Feb 04, 2008 at 09:00:34AM +0000, scott.postfix@... wrote:
                            >
                            >> >>To send DOS a textfile from a UNIX system use a DOS/UNIX conversion
                            >> >>tool or send the file as a base64 encoded attachment.
                            >>
                            >> Let me try to put it another way. How do I know when I need to convert the
                            >> file before I send it ? I don't intend to manually check every single text
                            >> file that I send to see if I should then manually convert it first.
                            >
                            >If you are sending data files (binary FTP over email), you don't convert
                            >at all, just base64 encode the file bits and send. If you are sending text
                            >files (ASCII FTP over email), and your text files are not actually UNIX
                            >text files, you need to convert them to unix text files before sending.
                            >
                            >> If this is correct, then shouldn't the MTA handle this mime type without
                            >> screwing up the file ?
                            >
                            >MTAs transmit messages, MIME encoding is the responsibility of an *MUA*.
                            >Postfix is not a MUA.
                            >
                            >--
                            > Viktor.
                            >
                            >Disclaimer: off-list followups get on-list replies or get ignored.
                            >Please do not ignore the "Reply-To" header.
                            >
                            >To unsubscribe from the postfix-users list, visit
                            >http://www.postfix.org/lists.html or click the link below:
                            ><mailto:majordomo@...?body=unsubscribe%20postfix-users>
                            >
                            >If my response solves your problem, the best way to thank me is to not
                            >send an "it worked, thanks" follow-up. If you must respond, please put
                            >"It worked, thanks" in the "Subject" so I can delete these quickly.
                          • Wietse Venema
                            ... No. The Postfix sendmail command expects input in UNIX TEXT format. If you provide something different, then all bets are off. ... begin speculation Your
                            Message 13 of 19 , Feb 4, 2008
                            • 0 Attachment
                              scott.postfix@...:
                              > Sorry for being a pain, but I want to get this clear. I can only see two
                              > things that could be wrong to get to the problem I am having.
                              >
                              > 1) Postfix is broken and should be detecting when text/plain attachments
                              > have CRLF and not add the CR.

                              No. The Postfix sendmail command expects input in UNIX TEXT format.
                              If you provide something different, then all bets are off.

                              > 2) My MUA (mutt) is broken and is incorrectly identifying a DOS text file
                              > as text/plain when it is not.

                              begin speculation

                              Your MUA (mutt) simply does what you told it to do. If you told it
                              to use Content-type: text/plain, while the input is not in UNIX
                              TEXT format, then all bets are off.

                              end speculation

                              The mere fact that some MTAs silently strip the extra CR does not
                              make it right.

                              Wietse
                            • Victor Duchovni
                              ... Postfix is an MTA. It is the MUA s responsibility to inject valid content. ... The MUA (mutt) is responsible for submitting valid content. What are you
                              Message 14 of 19 , Feb 4, 2008
                              • 0 Attachment
                                On Mon, Feb 04, 2008 at 02:57:38PM +0000, scott.postfix@... wrote:

                                > Sorry for being a pain, but I want to get this clear. I can only see two
                                > things that could be wrong to get to the problem I am having.
                                >
                                > 1) Postfix is broken and should be detecting when text/plain attachments
                                > have CRLF and not add the CR.

                                Postfix is an MTA. It is the MUA's responsibility to inject valid content.

                                > 2) My MUA (mutt) is broken and is incorrectly identifying a DOS text file
                                > as text/plain when it is not.

                                The MUA (mutt) is responsible for submitting valid content. What are you
                                doing with mutt, and what MIME content is mutt generating? Configure
                                mutt to use a custom "sendmail" that just saves the message to a disk
                                file. Report what you find in the disk file...

                                --
                                Viktor.

                                Disclaimer: off-list followups get on-list replies or get ignored.
                                Please do not ignore the "Reply-To" header.

                                To unsubscribe from the postfix-users list, visit
                                http://www.postfix.org/lists.html or click the link below:
                                <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                                If my response solves your problem, the best way to thank me is to not
                                send an "it worked, thanks" follow-up. If you must respond, please put
                                "It worked, thanks" in the "Subject" so I can delete these quickly.
                              • scott.postfix@scottrix.co.uk
                                ... Fair enough. ... A text file is produced with the email in it, the attachment is text/plain and the attachment has CRLF for each of it s lines. It seems
                                Message 15 of 19 , Feb 4, 2008
                                • 0 Attachment
                                  On Mon, Feb 04, 2008 at 10:10:43AM -0500, Victor Duchovni wrote:
                                  >On Mon, Feb 04, 2008 at 02:57:38PM +0000, scott.postfix@... wrote:
                                  >
                                  >> Sorry for being a pain, but I want to get this clear. I can only see two
                                  >> things that could be wrong to get to the problem I am having.
                                  >>
                                  >> 1) Postfix is broken and should be detecting when text/plain attachments
                                  >> have CRLF and not add the CR.
                                  >
                                  >Postfix is an MTA. It is the MUA's responsibility to inject valid content.

                                  Fair enough.

                                  >> 2) My MUA (mutt) is broken and is incorrectly identifying a DOS text
                                  >> file as text/plain when it is not.
                                  >
                                  >The MUA (mutt) is responsible for submitting valid content. What are you
                                  >doing with mutt, and what MIME content is mutt generating? Configure
                                  >mutt to use a custom "sendmail" that just saves the message to a disk
                                  >file. Report what you find in the disk file...

                                  A text file is produced with the email in it, the attachment is text/plain
                                  and the attachment has CRLF for each of it's lines.

                                  It seems the problem is with the MUA (mutt) and I believe that mutt gets
                                  the mime type from libmagic which also reports text/plain for the DOS text
                                  file. It can the tell the difference since "file dos.txt" gives

                                  "ASCII text, with CRLF line terminators"

                                  So I guess I can get it to give a different output for the mimetype too.

                                  Thanks for all your help with this,

                                  Scott.
                                • Noel Jones
                                  ... You re making this too complicated. The MUA software is expected to do the conversion for you. When an attachment is marked as text/plain it s expected
                                  Message 16 of 19 , Feb 4, 2008
                                  • 0 Attachment
                                    scott.postfix@... wrote:
                                    > On Mon, Feb 04, 2008 at 10:10:43AM -0500, Victor Duchovni wrote:
                                    >> On Mon, Feb 04, 2008 at 02:57:38PM +0000, scott.postfix@...
                                    >> wrote:
                                    >>
                                    >>> Sorry for being a pain, but I want to get this clear. I can only see
                                    >>> two things that could be wrong to get to the problem I am having.
                                    >>>
                                    >>> 1) Postfix is broken and should be detecting when text/plain
                                    >>> attachments have CRLF and not add the CR.
                                    >>
                                    >> Postfix is an MTA. It is the MUA's responsibility to inject valid
                                    >> content.
                                    >
                                    > Fair enough.
                                    >
                                    >>> 2) My MUA (mutt) is broken and is incorrectly identifying a DOS text
                                    >>> file as text/plain when it is not.
                                    >>
                                    >> The MUA (mutt) is responsible for submitting valid content. What are you
                                    >> doing with mutt, and what MIME content is mutt generating? Configure
                                    >> mutt to use a custom "sendmail" that just saves the message to a disk
                                    >> file. Report what you find in the disk file...
                                    >
                                    > A text file is produced with the email in it, the attachment is
                                    > text/plain and the attachment has CRLF for each of it's lines.
                                    >
                                    > It seems the problem is with the MUA (mutt) and I believe that mutt gets
                                    > the mime type from libmagic which also reports text/plain for the DOS
                                    > text file. It can the tell the difference since "file dos.txt" gives
                                    >
                                    > "ASCII text, with CRLF line terminators"
                                    >
                                    > So I guess I can get it to give a different output for the mimetype too.
                                    >
                                    > Thanks for all your help with this,
                                    >
                                    > Scott.

                                    You're making this too complicated. The MUA software is
                                    expected to do the conversion for you.

                                    When an attachment is marked as "text/plain" it's expected to
                                    be in the system's native encoding. The MUA on the receiving
                                    system will then alter the line ending as appropriate for the
                                    receiving system.

                                    When sending a text file from a Unix system, mime type
                                    text/plain must not end in CRLF.

                                    If you want to send a text attachment to an DOS/Windows
                                    system, just send it as a native Unix text file and the
                                    DOS/Windows MUA should present the attachment as a proper DOS
                                    text file. You need not bother with any utilities to
                                    add/remove the CR.

                                    If you need to send a file that is already encoded with CRLF
                                    on Unix, then it's not a Unix text file and shouldn't be sent
                                    as text/plain, and must be base64 encoded as a binary file.

                                    --
                                    Noel Jones
                                  • Victor Duchovni
                                    ... This is wrong. The message must either use CRLF for all message lines (RFC822 message) or LF for all message lines (unix message file). ... The MIME type
                                    Message 17 of 19 , Feb 4, 2008
                                    • 0 Attachment
                                      On Mon, Feb 04, 2008 at 03:59:38PM +0000, scott.postfix@... wrote:

                                      > >The MUA (mutt) is responsible for submitting valid content. What are you
                                      > >doing with mutt, and what MIME content is mutt generating? Configure
                                      > >mutt to use a custom "sendmail" that just saves the message to a disk
                                      > >file. Report what you find in the disk file...
                                      >
                                      > A text file is produced with the email in it, the attachment is text/plain
                                      > and the attachment has CRLF for each of it's lines.

                                      This is wrong. The message must either use CRLF for all message lines
                                      (RFC822 message) or LF for all message lines (unix message file).

                                      > It seems the problem is with the MUA (mutt) and I believe that mutt gets
                                      > the mime type from libmagic which also reports text/plain for the DOS text
                                      > file. It can the tell the difference since "file dos.txt" gives

                                      The MIME type is not the problem, mutt should be stripping the CR when
                                      encoding the text file into a message whose headers and body are LF
                                      terminated.

                                      --
                                      Viktor.

                                      Disclaimer: off-list followups get on-list replies or get ignored.
                                      Please do not ignore the "Reply-To" header.

                                      To unsubscribe from the postfix-users list, visit
                                      http://www.postfix.org/lists.html or click the link below:
                                      <mailto:majordomo@...?body=unsubscribe%20postfix-users>

                                      If my response solves your problem, the best way to thank me is to not
                                      send an "it worked, thanks" follow-up. If you must respond, please put
                                      "It worked, thanks" in the "Subject" so I can delete these quickly.
                                    Your message has been successfully submitted and would be delivered to recipients shortly.