Loading ...
Sorry, an error occurred while loading the content.

Possible bug in handling MIME messsage?

Expand Messages
  • herwighenseler
    Hi, I m using SOAP::Lite 0.60a on FreeBSD 5.2.1 and RedHat quite successful so far. But today I encountered a bug which I contribute to SOAP::Lite. I m sending
    Message 1 of 11 , Oct 22, 2004
    View Source
    • 0 Attachment
      Hi,

      I'm using SOAP::Lite 0.60a on FreeBSD 5.2.1 and RedHat quite
      successful so far. But today
      I encountered a bug which I contribute to SOAP::Lite.

      I'm sending files via a MIME-Message like this:

      my $ent = build MIME::Entity
      Type => "image/jpg",
      Encoding => "binary",
      Path => $path,
      'Content-Id'=> "<$cid>",
      'Connection'=> 'Keep-Alive',
      'Content-Length'=> -s $path,
      'Pragma' => 'no-cache',
      Disposition => "attachment";

      my $res = $self->{soap}->parts($ent)
      ->setFileMIME( -s $path, 'image/jpeg',
      SOAP::Data->name('data')->attr({'href'
      => "cid:$cid"}) );


      Works as advertised. But then I came across a file that ended with
      the character (hex) 0d.
      Turns out, that it was silently "dropped" and the resulting file was
      one byte shorter than
      expected. The terminating 0d was missing.

      I solved the problem by using "base64" encoding, but it seems to me
      that we have a bug in
      SOAP::Lite here which strips newlines too aggressive here. Or is the
      bug in MIME::Entity?
      Or (impossible, of course 8^) am I doing something wrong?

      TIA,
      Herwig
    • Byrne Reese
      SOAP::Lite uses MIME::Tools almost entirely - SOAP::Lite does no data management at all - it leaves that up to the MIME Parser. So I am hesitant to think
      Message 2 of 11 , Oct 22, 2004
      View Source
      • 0 Attachment
        SOAP::Lite uses MIME::Tools almost entirely - SOAP::Lite does no data
        management at all - it leaves that up to the MIME Parser. So I am
        hesitant to think SOAP::Lite is at the root.

        Try using SOAP::Lite 0.65 Beta 1 - the MIME stuff was substantially
        rearchitected there.

        You could try setting setting the 'Encoding' to base64 and see if
        MIME::Entity will encode it for you automagically...

        Keep me posted.

        Byrne

        herwighenseler wrote:

        >
        > Hi,
        >
        > I'm using SOAP::Lite 0.60a on FreeBSD 5.2.1 and RedHat quite
        > successful so far. But today
        > I encountered a bug which I contribute to SOAP::Lite.
        >
        > I'm sending files via a MIME-Message like this:
        >
        > my $ent = build MIME::Entity
        > Type => "image/jpg",
        > Encoding => "binary",
        > Path => $path,
        > 'Content-Id'=> "<$cid>",
        > 'Connection'=> 'Keep-Alive',
        > 'Content-Length'=> -s $path,
        > 'Pragma' => 'no-cache',
        > Disposition => "attachment";
        >
        > my $res = $self->{soap}->parts($ent)
        > ->setFileMIME( -s $path, 'image/jpeg',
        > SOAP::Data->name('data')->attr({'href'
        > => "cid:$cid"}) );
        >
        >
        > Works as advertised. But then I came across a file that ended with
        > the character (hex) 0d.
        > Turns out, that it was silently "dropped" and the resulting file was
        > one byte shorter than
        > expected. The terminating 0d was missing.
        >
        > I solved the problem by using "base64" encoding, but it seems to me
        > that we have a bug in
        > SOAP::Lite here which strips newlines too aggressive here. Or is the
        > bug in MIME::Entity?
        > Or (impossible, of course 8^) am I doing something wrong?
        >
        > TIA,
        > Herwig
        >
        >
        >
        >
        > *Yahoo! Groups Sponsor*
        > ADVERTISEMENT
        >
        >
      • Duncan Cameron
        ... Hmm ... A simple test using the new SOAP::Lite beta and MIME:Tools 6.200_02 seems to show that for a text file the trailing r n (0x0D 0x0A) is being
        Message 3 of 11 , Oct 22, 2004
        View Source
        • 0 Attachment
          At 2004-10-22, 17:29:30 Byrne Reese <byrne@...> wrote:
          >herwighenseler wrote:
          >>
          >> Hi,
          >>
          >> I'm using SOAP::Lite 0.60a on FreeBSD 5.2.1 and RedHat quite
          >> successful so far. But today
          >> I encountered a bug which I contribute to SOAP::Lite.
          >>
          >> I'm sending files via a MIME-Message like this:
          >>
          >> my $ent = build MIME::Entity
          >> Type => "image/jpg",
          >> Encoding => "binary",
          >> Path => $path,
          >> 'Content-Id'=> "<$cid>",
          >> 'Connection'=> 'Keep-Alive',
          >> 'Content-Length'=> -s $path,
          >> 'Pragma' => 'no-cache',
          >> Disposition => "attachment";
          >>
          >> my $res = $self->{soap}->parts($ent)
          >> ->setFileMIME( -s $path, 'image/jpeg',
          >> SOAP::Data->name('data')->attr({'href'
          >> => "cid:$cid"}) );
          >>
          >> Works as advertised. But then I came across a file that ended with
          >> the character (hex) 0d.
          >> Turns out, that it was silently "dropped" and the resulting file was
          >> one byte shorter than
          >> expected. The terminating 0d was missing.
          >>
          >> I solved the problem by using "base64" encoding, but it seems to me
          >> that we have a bug in
          >> SOAP::Lite here which strips newlines too aggressive here. Or is the
          >> bug in MIME::Entity?
          >> Or (impossible, of course 8^) am I doing something wrong?
          >>
          >> TIA,
          >> Herwig
          >>
          >SOAP::Lite uses MIME::Tools almost entirely - SOAP::Lite does no data
          >management at all - it leaves that up to the MIME Parser. So I am
          >hesitant to think SOAP::Lite is at the root.
          >
          >Try using SOAP::Lite 0.65 Beta 1 - the MIME stuff was substantially
          >rearchitected there.
          >
          >You could try setting setting the 'Encoding' to base64 and see if
          >MIME::Entity will encode it for you automagically...
          >
          >Keep me posted.
          >
          >Byrne

          Hmm ...

          A simple test using the new SOAP::Lite beta and MIME:Tools 6.200_02
          seems to show that for a text file the trailing \r\n (0x0D 0x0A) is
          being mangled into simply a \n (0x0A).

          The file being sent is

          line 1\r\n
          line 2\r\n
          line 3\r\n

          The file created by MIME::Parser (or a package that it calls) is

          line 1\r\n
          line 2\r\n
          line 3\n

          Another test with a file ending in \r (no trailing \n) has that
          character removed.

          Given these results I can imagine that it would easily have similar
          problems with binary files ending in \r.
          Does anyone have much experience of MIME::Tools to send attachments, as
          this looks like a pretty fundamental problem., or am I misunderstanding
          something?

          Regards
          Duncan
        • jpeyser
          The pod section of the MIME::Parser package states Fuzzing of CRLF and newline on output The 7bit and 8bit decoders will decode both a n and a r n
          Message 4 of 11 , Oct 22, 2004
          View Source
          • 0 Attachment
            The pod section of the MIME::Parser package states

            Fuzzing of CRLF and newline on output
            The "7bit" and "8bit" decoders will decode both a "\n"
            and a "\r\n" end-of-line sequence into a "\n".

            The "binary" decoder (default if no encoding specified)
            still outputs stuff verbatim... so a MIME message with
            CRLFs and no explicit encoding will be output as a text
            file that, on many systems, will have an annoying ^M at
            the end of each line... but this is as it should be.

            The SOAP::Packager::MIME package defaults the mime_encoding to '8bit'

            This has to be overridden with something like

            $soap->packager->{_content_encoding} = 'binary';

            Jonathan
          • Duncan Cameron
            I think that the problem is this line in MIME::Parser::Reader $last =~ s/[ r n]+ Z// if ($eos =~ /^(DELIM|CLOSE)/); It is trying to remove a trailing r n but,
            Message 5 of 11 , Oct 22, 2004
            View Source
            • 0 Attachment
              I think that the problem is this line in MIME::Parser::Reader

              $last =~ s/[\r\n]+\Z// if ($eos =~ /^(DELIM|CLOSE)/);

              It is trying to remove a trailing \r\n but, presumably due to the
              misusage of CRLF mentioned by Byrne, is also trying to handle other
              combinations of a line break, such as a \n on its own.

              This causes a \r at the end of the data to be removed.

              Regards
              Duncan

              ----- Original message -----

              From: Duncan Cameron <duncan_cameron2002@...>
              To: soaplite@yahoogroups.com <soaplite@yahoogroups.com>
              Sent: 2004-10-22, 17:12:23
              Subject: Re: [soaplite] Possible bug in handling MIME messsage?

              At 2004-10-22, 17:29:30 Byrne Reese <byrne@...> wrote:
              >herwighenseler wrote:
              >>
              >> Hi,
              >>
              >> I'm using SOAP::Lite 0.60a on FreeBSD 5.2.1 and RedHat quite
              >> successful so far. But today
              >> I encountered a bug which I contribute to SOAP::Lite.
              >>
              >> I'm sending files via a MIME-Message like this:
              >>
              >> my $ent = build MIME::Entity
              >> Type => "image/jpg",
              >> Encoding => "binary",
              >> Path => $path,
              >> 'Content-Id'=> "<$cid>",
              >> 'Connection'=> 'Keep-Alive',
              >> 'Content-Length'=> -s $path,
              >> 'Pragma' => 'no-cache',
              >> Disposition => "attachment";
              >>
              >> my $res = $self->{soap}->parts($ent)
              >> ->setFileMIME( -s $path, 'image/jpeg',
              >> SOAP::Data->name('data')->attr({'href'
              >> => "cid:$cid"}) );
              >>
              >> Works as advertised. But then I came across a file that ended with
              >> the character (hex) 0d.
              >> Turns out, that it was silently "dropped" and the resulting file was
              >> one byte shorter than
              >> expected. The terminating 0d was missing.
              >>
              >> I solved the problem by using "base64" encoding, but it seems to me
              >> that we have a bug in
              >> SOAP::Lite here which strips newlines too aggressive here. Or is the
              >> bug in MIME::Entity?
              >> Or (impossible, of course 8^) am I doing something wrong?
              >>
              >> TIA,
              >> Herwig
              >>
              >SOAP::Lite uses MIME::Tools almost entirely - SOAP::Lite does no data
              >management at all - it leaves that up to the MIME Parser. So I am
              >hesitant to think SOAP::Lite is at the root.
              >
              >Try using SOAP::Lite 0.65 Beta 1 - the MIME stuff was substantially
              >rearchitected there.
              >
              >You could try setting setting the 'Encoding' to base64 and see if
              >MIME::Entity will encode it for you automagically...
              >
              >Keep me posted.
              >
              >Byrne

              Hmm ...

              A simple test using the new SOAP::Lite beta and MIME:Tools 6.200_02
              seems to show that for a text file the trailing \r\n (0x0D 0x0A) is
              being mangled into simply a \n (0x0A).

              The file being sent is

              line 1\r\n
              line 2\r\n
              line 3\r\n

              The file created by MIME::Parser (or a package that it calls) is

              line 1\r\n
              line 2\r\n
              line 3\n

              Another test with a file ending in \r (no trailing \n) has that
              character removed.

              Given these results I can imagine that it would easily have similar
              problems with binary files ending in \r.
              Does anyone have much experience of MIME::Tools to send attachments, as
              this looks like a pretty fundamental problem., or am I misunderstanding
              something?

              Regards
              Duncan
            • Duncan Cameron
              ... I think what this means is that a text file sent as binary will not have the end of line sequence converted. This will be apparent on a Unix system (and
              Message 6 of 11 , Oct 22, 2004
              View Source
              • 0 Attachment
                On 2004-10-22 at 19:55:09 jpeyser wrote:

                >The pod section of the MIME::Parser package states
                >
                > Fuzzing of CRLF and newline on output
                > The "7bit" and "8bit" decoders will decode both a "\n"
                > and a "\r\n" end-of-line sequence into a "\n".
                >
                > The "binary" decoder (default if no encoding specified)
                > still outputs stuff verbatim... so a MIME message with
                > CRLFs and no explicit encoding will be output as a text
                > file that, on many systems, will have an annoying ^M at
                > the end of each line... but this is as it should be.
                >
                >The SOAP::Packager::MIME package defaults the mime_encoding to '8bit'
                >
                >This has to be overridden with something like
                >
                >$soap->packager->{_content_encoding} = 'binary';
                >
                >Jonathan

                I think what this means is that a 'text' file sent as binary will not
                have the end of line sequence converted. This will be apparent on a
                Unix system (and a Mac ?) by the extra ^M at the end of each line. It
                should not be an issue for Windows as the transmitted end of line
                sequence is that which is used for Windows.
                If the encoding is either 7bit or 8bit then the end of line should be
                converted to that appropriate for the receiving system. (But having now
                looked at some of MIME::Tools I would not be too sure of that).

                The original poster had specified binary anyway for the encoding in his
                example of losing a trailing \r.

                Regards
                Duncan
              • herwighenseler
                Hi, thanks for the responses so far. Yes, i tried with base64 encoding and yes, it worked. The trailing n was preserved. ... This could be the root of the
                Message 7 of 11 , Oct 26, 2004
                View Source
                • 0 Attachment
                  Hi,

                  thanks for the responses so far. Yes, i tried with "base64" encoding and yes, it worked.
                  The trailing \n was preserved.

                  --- In soaplite@yahoogroups.com, "jpeyser" <jpeyser@p...> wrote:
                  > The SOAP::Packager::MIME package defaults the mime_encoding to '8bit'
                  >
                  > This has to be overridden with something like
                  >
                  > $soap->packager->{_content_encoding} = 'binary';

                  This could be the root of the problem. Although I encoded my part with "binary"
                  (explicitly), the full MIME message seems to be encoded with "8-bit", causing problems.
                  Well, it should work since different parts of the message should be able to contain
                  different encoding messages, but I guess here is the problem. Still not sure if it's SOAP's
                  fault or MIME's.

                  I can't see how the last line should change the behaviour of the packager since
                  _content_encoding isn't used anywhere in SOAP::LIte. Aer you referring to SOAP::Lite 0,65
                  beta? (I'm using 0.60a).

                  Herwig
                • jpeyser
                  After much debuging, I think I found the problem. In Mime Tools module Mime/Parser/Reader.pm, at the end of sub read_chunk, the following line removes all
                  Message 8 of 11 , Oct 26, 2004
                  View Source
                  • 0 Attachment
                    After much debuging, I think I found the problem.

                    In Mime Tools module Mime/Parser/Reader.pm, at the end of sub
                    read_chunk, the following line removes all trailing \r and \n,
                    regardless of order.

                    ### Write out last held line, removing terminating CRLF if ended
                    on bound:
                    $last =~ s/[\r\n]+\Z// if ($eos =~ /^(DELIM|CLOSE)/);

                    It should read

                    $last =~ s/(?:\r\n)+\Z// if ($eos =~ /^(DELIM|CLOSE)/);

                    Jonathan
                  • Duncan Cameron
                    ... Indeed. I think that I reported this problem to the list earlier this week, and have also raised it on the CPAN bug tracking system. I think we need to be
                    Message 9 of 11 , Oct 26, 2004
                    View Source
                    • 0 Attachment
                      At 2004-10-26, 21:37:14 jpeyser <jpeyser@...> wrote:

                      >After much debuging, I think I found the problem.
                      >
                      >In Mime Tools module Mime/Parser/Reader.pm, at the end of sub
                      >read_chunk, the following line removes all trailing \r and \n,
                      >regardless of order.
                      >
                      > ### Write out last held line, removing terminating CRLF if ended
                      >on bound:
                      > $last =~ s/[\r\n]+\Z// if ($eos =~ /^(DELIM|CLOSE)/);
                      >
                      >It should read
                      >
                      > $last =~ s/(?:\r\n)+\Z// if ($eos =~ /^(DELIM|CLOSE)/);
                      Indeed.
                      I think that I reported this problem to the list earlier this week, and
                      have also raised it on the CPAN bug tracking system.

                      I think we need to be careful as to what the fix should be though. Your
                      example allows any number of \r\n sequences to be removed. I think that
                      only one of these should be removed

                      \r\n
                      \n

                      i.e. \r?\n
                      Because of the context of the code there should not be any other \n but
                      there can be any number of \r which should not be removed.
                      This allows for the sending system to have used either \r\n (correct)
                      or only \n (incorrect) as the line separator. MIME::Tools seems to be
                      full of code that expects only \n to be generated or to be received.

                      Regards
                      Duncan
                    • Byrne Reese
                      I am going to contact Eryq, the maintainer of MIME::Tools and see if I might be able to see what he thinks. He has helped SOAP::Lite in the past by issuing
                      Message 10 of 11 , Oct 26, 2004
                      View Source
                      • 0 Attachment
                        I am going to contact Eryq, the maintainer of MIME::Tools and see if I
                        might be able to see what he thinks. He has helped SOAP::Lite in the
                        past by issuing patches for us. He has good insight into the users of
                        MIME::Tools, the users of SOAP::Lite being a small minority. Let's get
                        his opinion.

                        =-D

                        Byrne

                        Duncan Cameron wrote:

                        > At 2004-10-26, 21:37:14 jpeyser <jpeyser@...> wrote:
                        >
                        > >After much debuging, I think I found the problem.
                        > >
                        > >In Mime Tools module Mime/Parser/Reader.pm, at the end of sub
                        > >read_chunk, the following line removes all trailing \r and \n,
                        > >regardless of order.
                        > >
                        > > ### Write out last held line, removing terminating CRLF if ended
                        > >on bound:
                        > > $last =~ s/[\r\n]+\Z// if ($eos =~ /^(DELIM|CLOSE)/);
                        > >
                        > >It should read
                        > >
                        > > $last =~ s/(?:\r\n)+\Z// if ($eos =~ /^(DELIM|CLOSE)/);
                        > Indeed.
                        > I think that I reported this problem to the list earlier this week, and
                        > have also raised it on the CPAN bug tracking system.
                        >
                        > I think we need to be careful as to what the fix should be though. Your
                        > example allows any number of \r\n sequences to be removed. I think that
                        > only one of these should be removed
                        >
                        > \r\n
                        > \n
                        >
                        > i.e. \r?\n
                        > Because of the context of the code there should not be any other \n but
                        > there can be any number of \r which should not be removed.
                        > This allows for the sending system to have used either \r\n (correct)
                        > or only \n (incorrect) as the line separator. MIME::Tools seems to be
                        > full of code that expects only \n to be generated or to be received.
                        >
                        > Regards
                        > Duncan
                        >
                      • jpeyser
                        Let me explain what is happening. First, the Reader uses n as the line terminator, so there will not be multiple n in one line. Second, the entity boundary
                        Message 11 of 11 , Oct 26, 2004
                        View Source
                        • 0 Attachment
                          Let me explain what is happening.

                          First, the Reader uses \n as the line terminator, so there will not be
                          multiple \n in one line. Second, the entity boundary is \r\n, i.e.
                          this is the terminator that MIME adds to the end of the entity. Third,
                          there is a multi-part boundary that begins with two dashes -> '--'.

                          So, when the Reader encounters the multipart boundary, it then checks
                          the last line to see if ends with the \r\n (the entity boundary). If so,
                          it removes it. (Yes, it should only remove it once.)

                          It's pretty straight forward. If windows is sending a CRLF as \n\r,
                          then the last line is \r\r\n. The incorrect code removes everything.
                          The correct code just removes the \r\n.

                          Jonathan
                        Your message has been successfully submitted and would be delivered to recipients shortly.