Loading ...
Sorry, an error occurred while loading the content.

On DKIM and Content-Transfer-Encoding

Expand Messages
  • Mauricio Tavares
    This email is mostly about understanding what is going on, how the process works than whether my install of postfix is working properly or the MUA is lying. In
    Message 1 of 7 , Jun 24, 2014
    • 0 Attachment
      This email is mostly about understanding what is going on, how
      the process works than whether my install of postfix is working
      properly or the MUA is lying. In fact, I am using ssmtp to send the
      email to postfix because I want to make my test as simple as possible
      (avoid "helpful" MUAs adding stuff to the email). So, let's say I
      create simple email:

      raub@desktop:~$ cat 8bit_test
      Subject: 8bit test
      Content-Transfer-Encoding: 8bit

      Text with umlauts here

      raub@desktop:~$

      It only has two headers (the From: header is added when I actually
      send email) and a single line in the body, which has a few 8bit
      characters. I am using ssmtp to send the above email to my postfix
      server,

      ssmtp -v raub@... < 8bit_test

      which is setup to do DKIM and announces that supports the following

      250-PIPELINING
      250-SIZE
      250-ETRN
      250-STARTTLS
      250-ENHANCEDSTATUSCODES
      250-8BITMIME
      250 DSN

      What I have noticed is that it will fail DKIM, and then will change
      the Content-Transfer-Encoding: header to quoted-printable. Now, if I
      submit the same email setting

      Content-Transfer-Encoding: quoted-printable

      it passes DKIM test. So, what is the 8bit encoding here anyway and how
      it compared to quoted-printable? From
      http://tools.ietf.org/html/rfc2045#page-14, it seems that
      quoted-printable converts the body to something 7bit can deal with (or
      at least the MTA will see that header and do the convertion). What
      about 8bit? Is it not translated? Or, is it expected to be fed in some
      kind of (mime) encoded format (hence the 8bitmime) so the MTA does not
      touch it (hence the mime processing controls mentioned in
      http://www.postfix.org/cleanup.8.html)?
    • Wietse Venema
      ... This email requires a MIME-Version header. Otherwise it is not valid MIME. ... If the client sends 8BITMIME, then it MUST issue the 8BITMIME keyword on the
      Message 2 of 7 , Jun 24, 2014
      • 0 Attachment
        Mauricio Tavares:
        > This email is mostly about understanding what is going on, how
        > the process works than whether my install of postfix is working
        > properly or the MUA is lying. In fact, I am using ssmtp to send the
        > email to postfix because I want to make my test as simple as possible
        > (avoid "helpful" MUAs adding stuff to the email). So, let's say I
        > create simple email:
        >
        > raub@desktop:~$ cat 8bit_test
        > Subject: 8bit test
        > Content-Transfer-Encoding: 8bit
        >
        > Text with umlauts here

        This email requires a MIME-Version header. Otherwise it is
        not valid MIME.

        > raub@desktop:~$
        >
        > It only has two headers (the From: header is added when I actually
        > send email) and a single line in the body, which has a few 8bit
        > characters. I am using ssmtp to send the above email to my postfix
        > server,
        >
        > ssmtp -v raub@... < 8bit_test
        >
        > which is setup to do DKIM and announces that supports the following
        >
        > 250-PIPELINING
        > 250-SIZE
        > 250-ETRN
        > 250-STARTTLS
        > 250-ENHANCEDSTATUSCODES
        > 250-8BITMIME
        > 250 DSN

        If the client sends 8BITMIME, then it MUST issue the 8BITMIME keyword
        on the MAIL FROM command line. Otherwise, behavior is undefined
        (i.e. different systems may handle this in different ways).

        > What I have noticed is that it will fail DKIM, and then will change
        > the Content-Transfer-Encoding: header to quoted-printable. Now, if I

        As required by the MIME RFCs, an MTA must either bounce mail or
        convert it to quoted-printable when it needs to deliver 8BITMIME
        mail to an SMTP server that does not announce 8BITMIME support.

        DKIM signatures of 8BITMIME mail may break unless all SMTP servers
        in the path implement and announce 8BITMIME support. Otherwise, it
        is better to down-convert to quoted-printable before DKIM signing.

        Wietse
      • Mauricio Tavares
        ... Thanks for the info. I then added MIME-Version: raub@desktop:~$ cat 8bit_test Subject: 8bit test MIME-Version: 1.0 Content-Transfer-Encoding: 8bit
        Message 3 of 7 , Jun 28, 2014
        • 0 Attachment
          On Tue, Jun 24, 2014 at 10:58 AM, Wietse Venema <wietse@...> wrote:
          > Mauricio Tavares:
          >> This email is mostly about understanding what is going on, how
          >> the process works than whether my install of postfix is working
          >> properly or the MUA is lying. In fact, I am using ssmtp to send the
          >> email to postfix because I want to make my test as simple as possible
          >> (avoid "helpful" MUAs adding stuff to the email). So, let's say I
          >> create simple email:
          >>
          >> raub@desktop:~$ cat 8bit_test
          >> Subject: 8bit test
          >> Content-Transfer-Encoding: 8bit
          >>
          >> Text with umlauts here
          >
          > This email requires a MIME-Version header. Otherwise it is
          > not valid MIME.
          >
          Thanks for the info. I then added MIME-Version:

          raub@desktop:~$ cat 8bit_test
          Subject: 8bit test
          MIME-Version: 1.0
          Content-Transfer-Encoding: 8bit

          Italienisches Olivenöl

          raub@desktop:~$

          and tried again using ssmtp. It still fails.

          >>
          >> It only has two headers (the From: header is added when I actually
          >> send email) and a single line in the body, which has a few 8bit
          >> characters. I am using ssmtp to send the above email to my postfix
          >> server,
          >>
          >> ssmtp -v raub@... < 8bit_test
          >>
          >> which is setup to do DKIM and announces that supports the following
          >>
          >> 250-PIPELINING
          >> 250-SIZE
          >> 250-ETRN
          >> 250-STARTTLS
          >> 250-ENHANCEDSTATUSCODES
          >> 250-8BITMIME
          >> 250 DSN
          >
          > If the client sends 8BITMIME, then it MUST issue the 8BITMIME keyword
          > on the MAIL FROM command line. Otherwise, behavior is undefined
          > (i.e. different systems may handle this in different ways).

          Let me use an example to see if I understand what you are
          saying: here is the transaction when I sent that email using ssmtp
          (which is talking to mail.domain.com, which is using postfix):

          ssmtp -v raub@... < 8bit_test.bare
          [<-] 220 mail.domain.com Test Mail Server
          [->] HELO raub.internal.domain.com
          [<-] 250 mail.domain.com
          [->] MAIL FROM:<raub@...>
          [<-] 250 2.1.0 Ok
          [->] RCPT TO:<raub@...>
          [<-] 250 2.1.5 Ok
          [->] DATA
          [<-] 354 End data with <CR><LF>.<CR><LF>
          [->] Received: by desktop.internal.domain.com (sSMTP sendmail
          emulation); Tue, 24 Jun 2014 17:05:06 -0400
          [->] From: "Mauricio Tavares" <raub@...>
          [->] Date: Tue, 24 Jun 2014 17:05:06 -0400
          [->] Subject: 8bit test - bare
          [->] MIME-Version: 1.0
          [->] Content-Transfer-Encoding: 8bit
          [->] Content-Type: text/plain; charset=ISO-8859-1; format=flowed
          [->]
          [->] Italienisches Oliven�l
          [->]
          [->] Same text
          [->]
          [->] .
          [<-] 250 2.0.0 Ok: queued as CC9FE80042
          [->] QUIT
          [<-] 221 2.0.0 Bye

          the proper MAIL FROM command line should be
          (http://tools.ietf.org/html/rfc6152):

          MAIL FROM:<raub@...> BODY=8BITMIME

          If that is the case, I think that explains issues I observed with a
          few email clients, notably thunderbird. Instead of posting the problem
          I saw, I thought it would make sense for me to understand how this is
          supposed to work first. Now I can rerun thunderbird in debug mode and
          see if it is sending a proper MAIL FROM command line stating to use
          8BITMIME.

          >
          >> What I have noticed is that it will fail DKIM, and then will change
          >> the Content-Transfer-Encoding: header to quoted-printable. Now, if I
          >
          > As required by the MIME RFCs, an MTA must either bounce mail or
          > convert it to quoted-printable when it needs to deliver 8BITMIME
          > mail to an SMTP server that does not announce 8BITMIME support.
          >
          > DKIM signatures of 8BITMIME mail may break unless all SMTP servers
          > in the path implement and announce 8BITMIME support. Otherwise, it
          > is better to down-convert to quoted-printable before DKIM signing.
          >
          Can I down-convert the email in postfix before DKIM signing? I
          believe some email clients using this server are sending 8BITMIME
          emails without a properly created MAIL FROM command line. Since I
          cannot do the right thing and correct that at the MUA side, I would
          like to do the next best thing.

          > Wietse
        • Andreas Schulze
          ... depending on your shell it s possible the ö is encoded as 2 byte in UTF-8. so you may need a charset declaration, too. does your test pass if you simply
          Message 4 of 7 , Jun 29, 2014
          • 0 Attachment
            Mauricio Tavares:
            > Content-Transfer-Encoding: 8bit
            >
            > Italienisches Olivenöl
            depending on your shell it's possible the 'ö' is encoded as 2 byte in UTF-8.
            so you may need a charset declaration, too.

            does your test pass if you simply replace ö by oe ?
            that way you may check if you test the right way at all.

            Andreas
          • Wietse Venema
            ... You send HELO. That means you can only send 7-bit ASCII email. Please read RFC 5321 for the 7-bit requirement of SMTP. In order to send 8BIT mail over
            Message 5 of 7 , Jun 29, 2014
            • 0 Attachment
              Mauricio Tavares:
              > [<-] 220 mail.domain.com Test Mail Server
              > [->] HELO raub.internal.domain.com
              > [<-] 250 mail.domain.com
              > [->] MAIL FROM:<raub@...>

              You send HELO. That means you can only send 7-bit ASCII email.
              Please read RFC 5321 for the 7-bit requirement of SMTP.

              In order to send 8BIT mail over SMTP, the client must announce that
              it supports ESMTP, the server must announce that it supports 8BITMIME,
              and the client must issue the 8BITMIME parameter in the MAIL FROM
              command.

              Please read RFC 1869 for how to negotiate ESMTP.
              Please read RFC 1652 for how to negotiate 8BITMIME.

              > > As required by the MIME RFCs, an MTA must either bounce mail or
              > > convert it to quoted-printable when it needs to deliver 8BITMIME
              > > mail to an SMTP server that does not announce 8BITMIME support.
              > >
              > > DKIM signatures of 8BITMIME mail may break unless all SMTP servers
              > > in the path implement and announce 8BITMIME support. Otherwise, it
              > > is better to down-convert to quoted-printable before DKIM signing.
              > >
              > Can I down-convert the email in postfix before DKIM signing? I
              > believe some email clients using this server are sending 8BITMIME
              > emails without a properly created MAIL FROM command line. Since I
              > cannot do the right thing and correct that at the MUA side, I would
              > like to do the next best thing.

              You can force downconversion with a null content filter and by
              suppressing or ignoring the 8BITMIME server announcement.

              Untested example:

              /etc/postfix/master.cf:
              smtp .. .. .. .. .. .. .. smtpd
              -o content_filter=dummy:127.0.0.1:12345

              dummy .. .. .. .. .. .. .. smtp
              -o smtp_discard_ehlo_keywords=8bitmime,silent-discard

              127.0.0.1:12345 .. .. .. .. .. .. .. smtpd
              -o smtpd_authorized_xforward_hosts=127.0.0.1
              -o smtpd_client_restrictions=
              -o smtpd_helo_restrictions=
              -o smtpd_sender_restrictions=
              -o smtpd_relay_restrictions=
              -o smtpd_recipient_restrictions=permit_mynetworks,reject

              Once mail is received with the 127.0.0.1:12345 SMTP server, it will
              have been down-converted, provided that it was formatted properly,
              and that the proper ESMTP commands were used.

              Wietse
            • Mauricio Tavares
              ... So, based on those two RFCs, the transaction should look something like: raub@desktop:/tmp$ nc -t mail.domain.com 25 220 mail.domain.com Test Mail Server
              Message 6 of 7 , Jul 2, 2014
              • 0 Attachment
                On Sun, Jun 29, 2014 at 8:20 AM, Wietse Venema <wietse@...> wrote:
                > Mauricio Tavares:
                >> [<-] 220 mail.domain.com Test Mail Server
                >> [->] HELO raub.internal.domain.com
                >> [<-] 250 mail.domain.com
                >> [->] MAIL FROM:<raub@...>
                >
                > You send HELO. That means you can only send 7-bit ASCII email.
                > Please read RFC 5321 for the 7-bit requirement of SMTP.
                >
                > In order to send 8BIT mail over SMTP, the client must announce that
                > it supports ESMTP, the server must announce that it supports 8BITMIME,
                > and the client must issue the 8BITMIME parameter in the MAIL FROM
                > command.
                >
                > Please read RFC 1869 for how to negotiate ESMTP.
                > Please read RFC 1652 for how to negotiate 8BITMIME.
                >
                So, based on those two RFCs, the transaction should look something like:

                raub@desktop:/tmp$ nc -t mail.domain.com 25
                220 mail.domain.com Test Mail Server
                EHLO desktop.domain.com
                250-mail.domain.com
                250-PIPELINING
                250-SIZE
                250-ETRN
                250-STARTTLS
                250-ENHANCEDSTATUSCODES
                250-8BITMIME
                250 DSN
                MAIL FROM:<raub@...> BODY=8BITMIME
                250 2.1.0 Ok
                RCPT TO:<raub@...>
                250 2.1.5 Ok
                DATA
                354 End data with <CR><LF>.<CR><LF>
                From: "Mauricio Tavares" <raub@...>
                Subject: 8bit test - manual6
                Date: Mon, 30 Jun 2014 11:10:05 -0400
                MIME-Version: 1.0
                Content-Transfer-Encoding: 8bit

                Olivenöl

                .
                250 2.0.0 Ok: queued as 2BA1F80041
                QUIT
                221 2.0.0 Bye
                raub@desktop:/tmp$

                or am I missing something?

                >> > As required by the MIME RFCs, an MTA must either bounce mail or
                >> > convert it to quoted-printable when it needs to deliver 8BITMIME
                >> > mail to an SMTP server that does not announce 8BITMIME support.
                >> >
                >> > DKIM signatures of 8BITMIME mail may break unless all SMTP servers
                >> > in the path implement and announce 8BITMIME support. Otherwise, it
                >> > is better to down-convert to quoted-printable before DKIM signing.
                >> >
                >> Can I down-convert the email in postfix before DKIM signing? I
                >> believe some email clients using this server are sending 8BITMIME
                >> emails without a properly created MAIL FROM command line. Since I
                >> cannot do the right thing and correct that at the MUA side, I would
                >> like to do the next best thing.
                >
                > You can force downconversion with a null content filter and by
                > suppressing or ignoring the 8BITMIME server announcement.
                >
                > Untested example:
                >
                > /etc/postfix/master.cf:
                > smtp .. .. .. .. .. .. .. smtpd
                > -o content_filter=dummy:127.0.0.1:12345
                >
                > dummy .. .. .. .. .. .. .. smtp
                > -o smtp_discard_ehlo_keywords=8bitmime,silent-discard
                >
                > 127.0.0.1:12345 .. .. .. .. .. .. .. smtpd
                > -o smtpd_authorized_xforward_hosts=127.0.0.1
                > -o smtpd_client_restrictions=
                > -o smtpd_helo_restrictions=
                > -o smtpd_sender_restrictions=
                > -o smtpd_relay_restrictions=
                > -o smtpd_recipient_restrictions=permit_mynetworks,reject
                >
                > Once mail is received with the 127.0.0.1:12345 SMTP server, it will
                > have been down-converted, provided that it was formatted properly,
                > and that the proper ESMTP commands were used.
                >
                > Wietse
              • Wietse Venema
                ... That s correct (except I can t tell if nc sends lines ending with bare , or if it sends lines ending with as required by SMTP; Postfix will
                Message 7 of 7 , Jul 2, 2014
                • 0 Attachment
                  Mauricio Tavares:
                  > raub@desktop:/tmp$ nc -t mail.domain.com 25
                  > 220 mail.domain.com Test Mail Server
                  > EHLO desktop.domain.com
                  > 250-mail.domain.com
                  > 250-PIPELINING
                  > 250-SIZE
                  > 250-ETRN
                  > 250-STARTTLS
                  > 250-ENHANCEDSTATUSCODES
                  > 250-8BITMIME
                  > 250 DSN
                  > MAIL FROM:<raub@...> BODY=8BITMIME
                  > 250 2.1.0 Ok
                  > RCPT TO:<raub@...>
                  > 250 2.1.5 Ok
                  > DATA
                  > 354 End data with <CR><LF>.<CR><LF>
                  > From: "Mauricio Tavares" <raub@...>
                  > Subject: 8bit test - manual6
                  > Date: Mon, 30 Jun 2014 11:10:05 -0400
                  > MIME-Version: 1.0
                  > Content-Transfer-Encoding: 8bit
                  >
                  > Oliven?l
                  >
                  > .
                  > 250 2.0.0 Ok: queued as 2BA1F80041
                  > QUIT
                  > 221 2.0.0 Bye
                  > raub@desktop:/tmp$
                  >
                  > or am I missing something?

                  That's correct (except I can't tell if nc sends lines ending with
                  bare <LF>, or if it sends lines ending with <CR><LF> as required
                  by SMTP; Postfix will accept either so it does not matter here).

                  Now, you can try to use the null-filter trick and see what it
                  looks like after 8-to-7 downconversion.

                  BTW domain.com is a real site; use example.com, example.net etc.
                  for examples. There is an RFC on that topic.

                  Wietse
                Your message has been successfully submitted and would be delivered to recipients shortly.