Loading ...
Sorry, an error occurred while loading the content.
 

utf-8 support on win32 with default code page 1250

Expand Messages
  • Zbynek Winkler
    Hello! A long time ago I ve posted a bug on sourceforge about utf-8 support on WinXP with default code page 1250 (default for czech). I ve given up that time
    Message 1 of 3 , Feb 4, 2003
      Hello!

      A long time ago I've posted a bug on sourceforge about utf-8 support on
      WinXP with default code page 1250 (default for czech). I've given up
      that time because I got the feeling that I am not able to convince
      anybody that there *is* a bug (that was also the resolution for closing
      the bug) thinking that in the meantime someone will fix it anyway.

      Well I've tested the 6.1 and it was still there. I did a websearch about
      utf-8 problems in vim and found
      http://groups.yahoo.com/group/vim-multibyte/message/585 which described
      exactly the same problem I was having. Because I really needed the utf-8
      this time, I've decided to take the sources and give it a shot.

      Debuging the file saving process showed that the iconv.dll was not used
      for the conversion at all (it was not even looked for) so it made no
      difference if I had it installed or not. The line that decided that no
      iconv was needed is 3118 in fileio.c - wb_flags = get_fio_flags(fenc);.
      When changed to wb_flags = 0; iconv finally got used for (it actually
      complained if I have not installed it) and the conversion was correct!

      I am not really sure what I have changed so it would probably be a good
      idea if someone more familiar with the code reviewed it and found the
      real cause for the wrong behavior (this probably being just a hotfix).
      The conclusion I've made is, that the built-in process cannot handle
      conversion from cp1250 to utf-8, so it should handle it to the iconv.dll.

      If you want me to do some more testing, please do not hesitate to
      contact me.

      Zbynek Winkler

      --
      <zw at matfyz.cz>
      http://zw.matfyz.cz/ http://robotika.cz/
      Faculty of Mathematics and Physics, Charles University, Prague, Czech
      Republic
    • Glenn Maynard
      ... iconv should not be used in Windows for codepages at all. Rather, MultiByteToWideChar and WideCharToMultiByte should be used. I think code to do this for
      Message 2 of 3 , Feb 4, 2003
        On Tue, Feb 04, 2003 at 08:14:25PM +0100, Zbynek Winkler wrote:
        > Debuging the file saving process showed that the iconv.dll was not used
        > for the conversion at all (it was not even looked for) so it made no
        > difference if I had it installed or not. The line that decided that no
        > iconv was needed is 3118 in fileio.c - wb_flags = get_fio_flags(fenc);.
        > When changed to wb_flags = 0; iconv finally got used for (it actually
        > complained if I have not installed it) and the conversion was correct!

        iconv should not be used in Windows for codepages at all. Rather,
        MultiByteToWideChar and WideCharToMultiByte should be used. I think
        code to do this for file I/O was integrated in CVS, but there may not
        have been a release since then, so give CVS a shot.

        --
        Glenn Maynard
      • Bram Moolenaar
        ... It looks like this problem was already fixed by patch 6.1.029. The code just before the line you changed now looks like this: if (converted && (enc_utf8
        Message 3 of 3 , Feb 5, 2003
          Zbynek Winkler wrote:

          > A long time ago I've posted a bug on sourceforge about utf-8 support on
          > WinXP with default code page 1250 (default for czech). I've given up
          > that time because I got the feeling that I am not able to convince
          > anybody that there *is* a bug (that was also the resolution for closing
          > the bug) thinking that in the meantime someone will fix it anyway.
          >
          > Well I've tested the 6.1 and it was still there. I did a websearch about
          > utf-8 problems in vim and found
          > http://groups.yahoo.com/group/vim-multibyte/message/585 which described
          > exactly the same problem I was having. Because I really needed the utf-8
          > this time, I've decided to take the sources and give it a shot.
          >
          > Debuging the file saving process showed that the iconv.dll was not used
          > for the conversion at all (it was not even looked for) so it made no
          > difference if I had it installed or not. The line that decided that no
          > iconv was needed is 3118 in fileio.c - wb_flags = get_fio_flags(fenc);.
          > When changed to wb_flags = 0; iconv finally got used for (it actually
          > complained if I have not installed it) and the conversion was correct!
          >
          > I am not really sure what I have changed so it would probably be a good
          > idea if someone more familiar with the code reviewed it and found the
          > real cause for the wrong behavior (this probably being just a hotfix).
          > The conclusion I've made is, that the built-in process cannot handle
          > conversion from cp1250 to utf-8, so it should handle it to the iconv.dll.
          >
          > If you want me to do some more testing, please do not hesitate to
          > contact me.

          It looks like this problem was already fixed by patch 6.1.029. The code
          just before the line you changed now looks like this:

          if (converted && (enc_utf8 || STRCMP(p_enc, "latin1") == 0))

          It used to be:

          if (converted && (enc_utf8 || !has_mbyte))

          Give that change a try, I guess it fixes your problem.

          --
          hundred-and-one symptoms of being an internet addict:
          137. You decide to stay in college for an additional year or two,
          just so you can have the free Internet access.

          /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
          /// Creator of Vim - Vi IMproved -- http://www.Vim.org \\\
          \\\ Project leader for A-A-P -- http://www.A-A-P.org ///
          \\\ Help AIDS victims, buy at Amazon -- http://ICCF.nl/click1.html ///
        Your message has been successfully submitted and would be delivered to recipients shortly.