Loading ...
Sorry, an error occurred while loading the content.

patch 838 has been mangled by improper recoding: problem investigation and proposed solutions [was: macosx x11/motif vim-7.3.905 test92 fails]

Expand Messages
  • Roland Eggner
    Content ... (1) Problem investigation (2) Proposed solutions (2.1) @ (2.2) @ Bram Hi ! ... Thank you for detailed report. (1)
    Message 1 of 11 , Apr 18, 2013
    • 0 Attachment
      Content
      -------
      (1) Problem investigation
      (2) Proposed solutions
      (2.1) @ <raf@...>
      (2.2) @ Bram


      Hi <raf@...>!

      On 2013-04-18 Thursday at 13:28 +1000 <raf@...> wrote:
      > Roland Eggner wrote:
      > > On 2013-04-17 Wednesday at 12:39 +1000 <raf@...> wrote:
      > > > … …
      > > > test test92.in file looks wrong here. perhaps multiply-encoded.
      > > > i've attached it.
      > > … …
      > > Hi <raf@...>!
      > >
      > > Thank you for your bugreport.
      > >
      > > When Bram and I wrote test92, I verified it to work with all 3 of utf8, latin
      > > and C locale. I did not take into account the case of a system providing no
      > > utf8-locale at all (would be considered a bit outdated in 21st century ..).
      >
      > my system (macosx-10.6.8) does provide utf8 locales.
      >
      > > If
      > > there is at least one such vim-using system, I will try to support it, probably
      > > by just skipping test92.
      >
      > i think there's an encoding problem in the test92.in file itself.
      > i downloaded patches from ftp.vim.org. it seems that 7.3.838 is
      > badly encoded. i've attached it. something like this has happened
      > before but only for patches on ftp.vim.org itself. other archives
      > weren't affected.
      >
      > > test93 succeeds for you?
      >
      > yes, it does.
      >
      > > If you want to help further, please provide the output of
      > > :view src/testdir/test92.in
      >
      > vim: set ft=vim fenc=utf-8:
      >
      > Tests if :mksession saves cursor columns correctly in presence of tab and
      > multibyte characters when fileencoding=utf-8.
      >
      > STARTTEST
      > :so mbyte.vim
      > :if !has('mksession')
      > : e! test.ok
      > : wq! test.out
      > :endif
      > :set sessionoptions=buffers splitbelow fileencoding=utf-8
      > /^start:
      > :vsplit
      > j16|:split
      > j16|:split
      > j16|:split
      > j8|:split
      > j8|:split
      > j16|:split
      > j16|:split
      > j16|:wincmd l
      > /^start:
      > :set nowrap
      > j16|3zl:split
      > j016|3zl:split
      > j016|3zl:split
      > j08|3zl:split
      > j08|3zl:split
      > j016|3zl:split
      > j016|3zl:split
      > j016|3zl:split
      > :mksession! test.out
      > :new test.out
      > :v/\(^ *normal! 0\|^ *exe 'normal!\)/d
      > :w
      > :qa!
      > ENDTEST
      >
      > start:
      > no multibyte chAracter
      > one leaDing tab
      > four leadinG spaces
      > two consecutive tabs
      > two tabs in one line
      > one … multibyteCharacter
      > a “b†two multiByte characters
      > “c†1€ three mulTibyte characters
      >
      > > :set fileencodings? encoding? fileencoding? termencoding?
      >
      > fileencodings=utf-8,latin1
      > encoding=utf-8
      > fileencoding=utf-8
      > termencoding=utf-8
      >
      > > :!locale
      >
      > LANG="en_AU.UTF-8"
      > LC_COLLATE="en_AU.UTF-8"
      > LC_CTYPE="en_AU.UTF-8"
      > LC_MESSAGES="en_AU.UTF-8"
      > LC_MONETARY="en_AU.UTF-8"
      > LC_NUMERIC="en_AU.UTF-8"
      > LC_TIME="en_AU.UTF-8"
      > LC_ALL=
      >
      > > :!locale -a
      >
      > i've skipped all of the non-english ones.
      > there are many.
      >
      > en_AU
      > en_AU.ISO8859-1
      > en_AU.ISO8859-15
      > en_AU.US-ASCII
      > en_AU.UTF-8
      > en_CA
      > en_CA.ISO8859-1
      > en_CA.ISO8859-15
      > en_CA.US-ASCII
      > en_CA.UTF-8
      > en_GB
      > en_GB.ISO8859-1
      > en_GB.ISO8859-15
      > en_GB.US-ASCII
      > en_GB.UTF-8
      > en_IE
      > en_IE.UTF-8
      > en_NZ
      > en_NZ.ISO8859-1
      > en_NZ.ISO8859-15
      > en_NZ.US-ASCII
      > en_NZ.UTF-8
      > en_US
      > en_US.ISO8859-1
      > en_US.ISO8859-15
      > en_US.US-ASCII
      > en_US.UTF-8
      > C
      > POSIX

      Thank you for detailed report.


      (1) Problem investigation
      --------------------------

      (1.1) Looks everything ok on your side, apart from file “test92.in”, which gots
      mangled by improper recoding _prior_ to upload to
      ftp://ftp.de.vim.org/patches/7.3/7.3.838
      and _prior_ to generating of
      ftp://ftp.de.vim.org/patches/7.3/MD5SUMS

      Currently I have no network access to ftp://ftp.vim.org/ … server seems to be
      overloaded or down, or DNS resolving to “ftp.nluug.nl” has been wrong.

      (1.2) Patch 7.3.838 exhibits a problem, which mostly does not matter, but in
      this case it does: Patch files should generally, and this one _needs_ to be
      treated as binary file. This means for transmission e.g. mimetype
      “application/octet-stream”, and within vim e.g. “:view ++binary 7.3.838.patch”.

      Purpose of the patch is to test vim handling of text files with _different_
      encodings on :mksession, and thus contains by design one part, which could be
      printed after decoding from binary to utf-8, and another part, which could be
      printed after decoding from binary to ISO-8859-1. Decoding and printing of both
      parts _together_ is impossible, instead the patch file needs to be treated as
      binary.

      (1.3) I must admit, even when I posted the latest version of the patch to the
      mailing list, the attachment holding the patch has been declared wrongly as
      “Content-Type: text/plain; charset=utf-8“. Even so, we could say “by good
      luck”, it hit the mailing list correctly, as I have verified after downloading
      from
      http://article.gmane.org/gmane.editors.vim.devel/37751

      (1.4) In mercurial repository the patch has been committed correctly, as I have
      verified after (pseudocode)
      hg clone -U http://vim.googlecode.com/hg/ vim
      cd vim
      hg log --git --patch --rev 1c819b05529cbbc39982513271499e0217382124

      (1.5) Within transmission from mercurial repository to the mailing list the
      patch file gots mangled by improper recoding, as I have found in my mail archive
      > From: Bram Moolenaar <Bram@...>
      > Subject: Patch 7.3.838
      > Date: Tue, 26 Feb 2013 17:23:01 +0100
      … and after downloading from
      http://article.gmane.org/gmane.editors.vim.devel/37820


      (2) Proposed solutions
      -----------------------

      (2.1) @ <raf@...>
      ----------------------
      If you want to track “bleeding edge” versions of vim:
      Use mercurial, e.g. “hg clone http://vim.googlecode.com/hg/ vim”. As an
      additional goody this would save you troubles resulting from inconsistent source
      tree, because so far patches published on ftp://ftp.vim.org/ and its mirrors
      exclude everything in and below source directory “runtime”.

      Otherwise, you could download the patch from URL mentioned under (1.3).

      Or you could use file “test92.in” attached to this mail.

      (2.2) @ Bram
      -------------
      (2.2.1) ftp://ftp.de.vim.org/patches/7.3/README already mentiones a few
      patches, which required kind of “extra treatment”. IHMO a similar solution for
      patch 7.3.838 would help vim users of this ftp directory; you could generate
      the fixed patch by e.g.
      hg export --git --rev 1c819b05529cbbc39982513271499e0217382124

      (2.2.2) Vim provides filetype detection and syntax highlighting for patch
      files. Maybe, vim should automatically switch on option binary on detection of
      patch files, instead of requiring users to think of option “++binary”?

      --
      Regards
      Roland Eggner
    • Roland Eggner
      ... Wrong file attached, sorry. Here is the intended “test92.in”. -- Regards Roland Eggner
      Message 2 of 11 , Apr 18, 2013
      • 0 Attachment
        On 2013-04-18 Thursday at 19:36 +0200 Roland Eggner wrote:
        > Content
        > -------
        > (1) Problem investigation
        > (2) Proposed solutions
        > (2.1) @ <raf@...>
        > (2.2) @ Bram

        Wrong file attached, sorry. Here is the intended “test92.in”.

        --
        Regards
        Roland Eggner
      • Bram Moolenaar
        ... [...] ... I have fixed the patch file now. No idea how this happened. Even more strange, the test92.in file went missing in my source directory. This
        Message 3 of 11 , Apr 18, 2013
        • 0 Attachment
          Roland Eggner wrote:

          > Content
          > -------
          > (1) Problem investigation
          > (2) Proposed solutions
          > (2.1) @ <raf@...>
          > (2.2) @ Bram

          [...]

          > (1) Problem investigation
          > --------------------------
          >
          > (1.1) Looks everything ok on your side, apart from file “test92.in”, which gots
          > mangled by improper recoding _prior_ to upload to
          > ftp://ftp.de.vim.org/patches/7.3/7.3.838
          > and _prior_ to generating of
          > ftp://ftp.de.vim.org/patches/7.3/MD5SUMS

          I have fixed the patch file now. No idea how this happened.

          Even more strange, the test92.in file went missing in my source
          directory. This doesn't make the tests fail, unfortunately.


          --
          I'm in shape. Round IS a shape.

          /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
          /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
          \\\ an exciting new programming language -- http://www.Zimbu.org ///
          \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • Ken Takata
          Hi Bram, ... With the new patch, test93.in seems to be encoded in utf-8 (not latin-1). Is it OK? Thanks, Ken Takata -- -- You received this message from the
          Message 4 of 11 , Apr 18, 2013
          • 0 Attachment
            Hi Bram,

            Bram Moolenaar wrote:

            > I have fixed the patch file now. No idea how this happened.
            >
            > Even more strange, the test92.in file went missing in my source
            > directory. This doesn't make the tests fail, unfortunately.

            With the new patch, test93.in seems to be encoded in utf-8 (not latin-1).
            Is it OK?

            Thanks,
            Ken Takata

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          • Roland Eggner
            ... Hi Ken, thank you for taking care. You are right, encoding of “test93.in” should be latin-1, not utf-8. Bram fixed the first, but missed the second
            Message 5 of 11 , Apr 19, 2013
            • 0 Attachment
              On 2013-04-18 Thursday at 15:32 -0700 Ken Takata wrote:
              > Bram Moolenaar wrote:
              > > I have fixed the patch file now. No idea how this happened.
              > >
              > > Even more strange, the test92.in file went missing in my source
              > > directory. This doesn't make the tests fail, unfortunately.
              >
              > With the new patch, test93.in seems to be encoded in utf-8 (not latin-1).
              > Is it OK?

              Hi Ken,

              thank you for taking care. You are right, encoding of “test93.in” should be
              latin-1, not utf-8. Bram fixed the first, but missed the second non-ascii hunk
              in patch 7.3.838.


              Hi Bram,

              hg export --git --rev 1c819b05529cbbc39982513271499e0217382124
              and
              hg log --git --patch --rev 1c819b05529cbbc39982513271499e0217382124
              both don't give what you want? Maybe attached tar.bz2 is better help for you?
              Just extract, check, and if ok for you, upload the 4 files.


              tar -tjvf patch-7.3.838.update_README_MD5SUMS.tar.bz2
              -----------------------------------------------------
              -rw------- roland/roland 11082 2013-02-26 15:23 patch-7.3.838.update/7.3.838
              -rw------- roland/roland 11057 2013-04-19 21:30 patch-7.3.838.update/7.3.838.updated
              -rw------- roland/roland 38060 2013-04-19 22:58 patch-7.3.838.update/MD5SUMS
              -rw------- roland/roland 60357 2013-04-19 22:51 patch-7.3.838.update/README


              (1) patch-7.3.838.update/7.3.838
              The mangled version as published 2013-02-26. Publishing can never be “undone”.
              Any trials to “undo” publishing would yield just the opposit of the wanted
              effect. Thus, I would restore the first published version.


              (2) patch-7.3.838.update/7.3.838.updated
              New filename helps you, all other vim developers, and all users of the ftp
              directory to distinguish easily and reliably the fixed from the mangled version
              of the patch. Additionally it saves problems related to proxy servers, and it
              saves problems related to search engines.


              (3) patch-7.3.838.update/MD5SUMS
              Entry for (1) already matches. Entry for (2) added.
              diff -u patch-7.3.838.update/MD5SUMS{~,}
              --- MD5SUMS~ 2013-04-15 19:23:31.000000000 +0200
              +++ MD5SUMS 2013-04-19 22:58:56.927657284 +0200
              @@ -836,6 +836,7 @@
              0eb860a53f948645085c5236b0819af4 7.3.836
              51f60b66613b7535855d1e592f544d83 7.3.837
              a3b9ad5d94ab29fd0891db19939a5bb8 7.3.838
              +3190f49784918e4e87dd678e5cb6265b 7.3.838.updated
              3a1e2a7ef2cb8c98a75e4121d58ebd59 7.3.839
              21e3dcb67ba9926ec7ab8b92c1c3869e 7.3.840
              ca0b0640f383d02715aa74ade316fbd9 7.3.841

              Please verify by something like:
              cd patch-7.3.838.update
              md5sum -c <( grep \ 7.3.838 MD5SUMS )
              7.3.838: OK
              7.3.838.updated: OK


              (4) patch-7.3.838.update/README
              diff -u patch-7.3.838.update/README{~,}
              --- README~ 2013-04-18 20:33:05.000000000 +0200
              +++ README 2013-04-19 22:51:33.377643391 +0200
              @@ -869,7 +869,8 @@
              1755 7.3.835 "xxd -i" fails on an empty file
              59985 7.3.836 clipboard does not work on Win32 when compiled with Cygwin
              2221 7.3.837 empty lines in :version output when 'columns' is 320
              - 11064 7.3.838 (after 7.3.830) insufficient testing for mksession
              + 11082 7.3.838 (obsolete, please use updated patch) insufficient testing for mksession
              + 11057 7.3.838.updated (after 7.3.830) insufficient testing for mksession
              2012 7.3.839 some files missing in the list of distributed files
              1652 7.3.840 "\@<!" in regexp does not work correctly with cp932
              2073 7.3.841 parse error in "cond ? one : two" expression with a subscript


              --
              Regards
              Roland Eggner
            • Bram Moolenaar
              ... No, these characters were also wrong. Update them now. -- hundred-and-one symptoms of being an internet addict: 194. Your business cards contain your
              Message 6 of 11 , Apr 20, 2013
              • 0 Attachment
                Ken Takata wrote:

                > Bram Moolenaar wrote:
                >
                > > I have fixed the patch file now. No idea how this happened.
                > >
                > > Even more strange, the test92.in file went missing in my source
                > > directory. This doesn't make the tests fail, unfortunately.
                >
                > With the new patch, test93.in seems to be encoded in utf-8 (not latin-1).
                > Is it OK?

                No, these characters were also wrong. Update them now.

                --
                hundred-and-one symptoms of being an internet addict:
                194. Your business cards contain your e-mail and home page address.

                /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                \\\ an exciting new programming language -- http://www.Zimbu.org ///
                \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                --
                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php

                ---
                You received this message because you are subscribed to the Google Groups "vim_dev" group.
                To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                For more options, visit https://groups.google.com/groups/opt_out.
              • Bram Moolenaar
                ... I already updated the file. I also changed the header now, although that s just for information, no tool should actually use it. -- Some of the well known
                Message 7 of 11 , Apr 20, 2013
                • 0 Attachment
                  Roland Eggner wrote:

                  > On 2013-04-18 Thursday at 15:32 -0700 Ken Takata wrote:
                  > > Bram Moolenaar wrote:
                  > > > I have fixed the patch file now. No idea how this happened.
                  > > >
                  > > > Even more strange, the test92.in file went missing in my source
                  > > > directory. This doesn't make the tests fail, unfortunately.
                  > >
                  > > With the new patch, test93.in seems to be encoded in utf-8 (not latin-1).
                  > > Is it OK?
                  >
                  > Hi Ken,
                  >
                  > thank you for taking care. You are right, encoding of “test93.in”
                  > should be latin-1, not utf-8. Bram fixed the first, but missed the
                  > second non-ascii hunk in patch 7.3.838.
                  >
                  >
                  > Hi Bram,
                  >
                  > hg export --git --rev 1c819b05529cbbc39982513271499e0217382124
                  > and
                  > hg log --git --patch --rev 1c819b05529cbbc39982513271499e0217382124
                  > both don't give what you want? Maybe attached tar.bz2 is better help
                  > for you? Just extract, check, and if ok for you, upload the 4 files.
                  >
                  >
                  > tar -tjvf patch-7.3.838.update_README_MD5SUMS.tar.bz2
                  > -----------------------------------------------------
                  > -rw------- roland/roland 11082 2013-02-26 15:23 patch-7.3.838.update/7.3.838
                  > -rw------- roland/roland 11057 2013-04-19 21:30 patch-7.3.838.update/7.3.838.updated
                  > -rw------- roland/roland 38060 2013-04-19 22:58 patch-7.3.838.update/MD5SUMS
                  > -rw------- roland/roland 60357 2013-04-19 22:51 patch-7.3.838.update/README
                  >
                  >
                  > (1) patch-7.3.838.update/7.3.838
                  > The mangled version as published 2013-02-26. Publishing can never be “undone”.
                  > Any trials to “undo” publishing would yield just the opposit of the wanted
                  > effect. Thus, I would restore the first published version.
                  >
                  >
                  > (2) patch-7.3.838.update/7.3.838.updated
                  > New filename helps you, all other vim developers, and all users of the ftp
                  > directory to distinguish easily and reliably the fixed from the
                  > mangled version of the patch. Additionally it saves problems related
                  > to proxy servers, and it saves problems related to search engines.

                  I already updated the file. I also changed the header now, although
                  that's just for information, no tool should actually use it.

                  --
                  Some of the well known MS-Windows errors:
                  EHUH Unexpected error
                  EUSER User error, not our fault!
                  EGOD Horrible problem, god knows what has happened
                  EERR Errornous error: nothing wrong

                  /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                  /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                  \\\ an exciting new programming language -- http://www.Zimbu.org ///
                  \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                  --
                  --
                  You received this message from the "vim_dev" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php

                  ---
                  You received this message because you are subscribed to the Google Groups "vim_dev" group.
                  To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                  For more options, visit https://groups.google.com/groups/opt_out.
                Your message has been successfully submitted and would be delivered to recipients shortly.