Loading ...
Sorry, an error occurred while loading the content.

Re: Unicode conversion bug?

Expand Messages
  • François Pinard
    [Tony Mechelynck] ... This is a common misconception. Unicode can represent many things, not anything. On one side, the W3C consortium has dispositions
    Message 1 of 25 , Mar 13, 2008
    • 0 Attachment
      [Tony Mechelynck]

      >[...] and since that Unicode encoding can represent anything [...]

      This is a common misconception. Unicode can represent many things, not
      anything. On one side, the W3C consortium has dispositions against
      attributing, in the future, single code points where combination
      characters would do, while keeping in Unicode what has already been
      lobbied by richer countries. That is, Unicode is meant to be easier to
      use for some than for others. Unicode is also set for supporting "main"
      scripts, not necessarily all of them. It means that poorer nations have
      less chance to get their script well represented in Unicode, if at all.

      --
      François Pinard http://pinard.progiciels-bpi.ca

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_multibyte" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • za
      Did you come upon this kind of trouble? my GVIM lose syntax hightlighting oftenly. especially when I switch to another tab or buffer I checked the plugins i
      Message 2 of 25 , Mar 14, 2008
      • 0 Attachment
        Did you come upon this kind of trouble?

        my GVIM lose syntax hightlighting oftenly.
        especially when I switch to another tab or buffer
        I checked the plugins i used for filetypes or global, no extras besides
        native things there, or maybe I once put in and then forgot?

        by the way, I always edit .html, php, .c, .java files in same time, this
        seems to heavy the problem, which is really boreing to type ":syntax on
        " or even shorthand every minute. :(






        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_multibyte" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • Nico Weber
        ... Which platform are you using? Does it also happen if you start vim with `gvim -u NONE -U NONE`? That prevents that your .vimrc file is loaded. You can use
        Message 3 of 25 , Mar 14, 2008
        • 0 Attachment
          > Did you come upon this kind of trouble?
          >
          > my GVIM lose syntax hightlighting oftenly.
          > especially when I switch to another tab or buffer
          > I checked the plugins i used for filetypes or global, no extras
          > besides
          > native things there, or maybe I once put in and then forgot?
          >
          > by the way, I always edit .html, php, .c, .java files in same time,
          > this
          > seems to heavy the problem, which is really boreing to type
          > ":syntax on
          > " or even shorthand every minute. :(

          Which platform are you using? Does it also happen if you start vim
          with `gvim -u NONE -U NONE`? That prevents that your .vimrc file is
          loaded.

          You can use `:verbose set syntax` to see where 'syntax' was modified
          the last time. Perhaps this gives you a hint.

          HTH,
          Nico

          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_multibyte" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • za
          Thanks for the comments, I think this will be a long-term question to find the answer. the platform is Linux/ubuntu 7.10 desktop with ZH_CN.UTF-8 locale The
          Message 4 of 25 , Mar 23, 2008
          • 0 Attachment
            Thanks for the comments, I think this will be a long-term question to
            find the answer.
            the platform is Linux/ubuntu 7.10 desktop with ZH_CN.UTF-8 locale
            The gvim is start from console without parameters.

            the output of :verbose set syntax is:
            syntax=xhtml
            最近修改于 /usr/share/vim/vim71/syntax/syntax.vim

            when i lost syntax, i am editing HTML file contain javascript

            在 2008-03-14五的 16:10 +0100,Nico Weber写道:
            > > Did you come upon this kind of trouble?
            > >
            > > my GVIM lose syntax hightlighting oftenly.
            > > especially when I switch to another tab or buffer
            > > I checked the plugins i used for filetypes or global, no extras
            > > besides
            > > native things there, or maybe I once put in and then forgot?
            > >
            > > by the way, I always edit .html, php, .c, .java files in same time,
            > > this
            > > seems to heavy the problem, which is really boreing to type
            > > ":syntax on
            > > " or even shorthand every minute. :(
            >
            > Which platform are you using? Does it also happen if you start vim
            > with `gvim -u NONE -U NONE`? That prevents that your .vimrc file is
            > loaded.
            >
            > You can use `:verbose set syntax` to see where 'syntax' was modified
            > the last time. Perhaps this gives you a hint.
            >
            > HTH,
            > Nico
            >
            > >
            >



            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_multibyte" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • T.P.S.Nakagawa
            This is Nakagawa, Japanese user on unix and Windows XP SP2. ( Windows Japanese version s native character set is cp932 ) I found same problem 30 April, another
            Message 5 of 25 , May 1, 2008
            • 0 Attachment
              This is Nakagawa, Japanese user on unix and Windows XP SP2.

              ( Windows Japanese version's native character set is cp932 )


              I found same problem 30 April, another way.
              ( I did'n t read this ML usually )


              I found.

              Windows notepad can read no-bomb UTF-8 file, but if edited or created
              file is saved with bomb.

              By unicode consocium standard, UTF-8 file can added bomb.
              but UTF-8 not need bomb.


              Vim use libiconv by FSF.
              I search on unix platform, `man iconv_open`, libiconv don't know
              UTF-8 with bomb. Then, libiconv can't detect UTF-8 with bomb.
              this is problem of libiconv, not Vim, by strict logic.


              In this reason, In my vimrc set filencodings sorted earlier UTF-8 than cp932
              but notepad saved file detected cp932.

              On justice way, We have to patch to libiconv, add UTF-8 bomb.
              But Can't save this weak point by Vim side?


              --
              Nakagawa Tsuneo mailto:yaemon@...
              Web site ( Jpanese only ) http://www.kikansha.jp/~yaemon/


              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_multibyte" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • Tony Mechelynck
              ... The BOM is a valid codepoint, ZERO-WIDTH NO-BREAK SPACE; however its use in that capacity is now deprecated (ZERO-WIDTH NON-JOINER is preferred IIUC).
              Message 6 of 25 , May 1, 2008
              • 0 Attachment
                On 02/05/08 03:29, T.P.S.Nakagawa wrote:
                > This is Nakagawa, Japanese user on unix and Windows XP SP2.
                >
                > ( Windows Japanese version's native character set is cp932 )
                >
                >
                > I found same problem 30 April, another way.
                > ( I did'n t read this ML usually )
                >
                >
                > I found.
                >
                > Windows notepad can read no-bomb UTF-8 file, but if edited or created
                > file is saved with bomb.
                >
                > By unicode consocium standard, UTF-8 file can added bomb.
                > but UTF-8 not need bomb.
                >
                >
                > Vim use libiconv by FSF.
                > I search on unix platform, `man iconv_open`, libiconv don't know
                > UTF-8 with bomb. Then, libiconv can't detect UTF-8 with bomb.
                > this is problem of libiconv, not Vim, by strict logic.
                >
                >
                > In this reason, In my vimrc set filencodings sorted earlier UTF-8 than cp932
                > but notepad saved file detected cp932.
                >
                > On justice way, We have to patch to libiconv, add UTF-8 bomb.
                > But Can't save this weak point by Vim side?
                >
                >

                The BOM is a valid codepoint, ZERO-WIDTH NO-BREAK SPACE; however its use
                in that capacity is now deprecated (ZERO-WIDTH NON-JOINER is preferred
                IIUC). Still, if any software accepts UTF-8 files "only" if there is no
                BOM in them, that software cannot be called UTF-8-compliant.

                I don't know which libiconv you are using, but my version of Vim (which
                has +iconv and a GTK2/Gnome2 GUI) accepts the BOM with no problem, not
                only in UTF-16/UTF-32 but also in UTF-8.


                Best regards,
                Tony.
                --
                The church is near but the road is icy; the bar is far away but I will
                walk carefully.
                -- Russian Proverb

                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_multibyte" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • T.P.S.Nakagawa
                Thanks for reading my broken english, and retry and reply. ... ... Sorry , I did t try by gvim on Windows ( downloaded, version 7.1 / 2007 May 12 compiled
                Message 7 of 25 , May 2, 2008
                • 0 Attachment
                  Thanks for reading my broken english, and retry and reply.


                  2008-05-02 11:36 (JST) , Tony Mechelynck sent follow message:
                  >> Vim use libiconv by FSF.
                  >> I search on unix platform, `man iconv_open`, libiconv don't know
                  >> UTF-8 with bomb. Then, libiconv can't detect UTF-8 with bomb.
                  >> this is problem of libiconv, not Vim, by strict logic.
                  >>
                  >>
                  >> In this reason, In my vimrc set filencodings sorted earlier UTF-8 than cp932
                  >> but notepad saved file detected cp932.
                  >>
                  >> On justice way, We have to patch to libiconv, add UTF-8 bomb.
                  >> But Can't save this weak point by Vim side?
                  >>
                  >>
                  <>
                  > I don't know which libiconv you are using, but my version of Vim (which
                  > has +iconv and a GTK2/Gnome2 GUI) accepts the BOM with no problem, not
                  > only in UTF-16/UTF-32 but also in UTF-8.
                  >

                  Sorry , I did't try by gvim on Windows ( downloaded, version 7.1 / 2007 May 12
                  compiled by Mr. Bram ) + related by http://www.vim.org/download.php#pc
                  old version sourceforge libiconv for Win32
                  http://sourceforge.net/project/showfiles.php?group_id=25167&package_id=51458
                  ( Release 1.9.1 / January 14 2004 )

                  on unix ( FreeBSD 5-stable ), with libiconv version 1.11 (+ OS patch 1 )
                  same (.|_)vimrc file, console vim can detect UTF-8 + BOM by notepad.
                  ( vim version is self compile by cvs source 2007 July 31, he say version 7.1.147 )


                  Someone compile libiconv ver 1.11 or later for Win 32 platform?
                  ( I tried 6 month ago, and I fail and busy )

                  --
                  Nakagawa Tsuneo mailto:yaemon@...
                  Web site ( Jpanese only ) http://www.kikansha.jp/~yaemon/


                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_multibyte" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                • Tony Mechelynck
                  On 02/05/08 10:17, T.P.S.Nakagawa wrote: [...] ... This is 7.1.000. I recommend the updated Vim and gvim compiled by Steve Hall (currently 7.1.293), obtainable
                  Message 8 of 25 , May 2, 2008
                  • 0 Attachment
                    On 02/05/08 10:17, T.P.S.Nakagawa wrote:
                    [...]
                    > Sorry , I did't try by gvim on Windows ( downloaded, version 7.1 / 2007 May 12
                    > compiled by Mr. Bram ) + related by http://www.vim.org/download.php#pc
                    > old version sourceforge libiconv for Win32
                    > http://sourceforge.net/project/showfiles.php?group_id=25167&package_id=51458
                    > ( Release 1.9.1 / January 14 2004 )

                    This is 7.1.000. I recommend the updated Vim and gvim compiled by Steve
                    Hall (currently 7.1.293), obtainable from
                    https://sourceforge.net/project/showfiles.php?group_id=43866&package_id=39721
                    . It has +iconv/dyn, its full ":version" text can be seen by clicking on
                    the word "Notes" or the clipboard-like icon next to the version number
                    on that same page.

                    >
                    > on unix ( FreeBSD 5-stable ), with libiconv version 1.11 (+ OS patch 1 )
                    > same (.|_)vimrc file, console vim can detect UTF-8 + BOM by notepad.
                    > ( vim version is self compile by cvs source 2007 July 31, he say version 7.1.147 )
                    >
                    >
                    > Someone compile libiconv ver 1.11 or later for Win 32 platform?
                    > ( I tried 6 month ago, and I fail and busy )
                    >

                    Libiconv 1.9.2 is available precompiled from GnuWin32 from the page
                    http://gnuwin32.sourceforge.net/packages/libiconv.htm . This is one
                    sub-sub-level above what you already have -- you might try using this
                    until or unless you succeed to get (or make) a later version. Since
                    Steve Hall's builds include +iconv/dyn and the "Compilation" and
                    "Linking" parts of its ":version" text mention no iconv version, I
                    suppose upgrading iconv (if and when) means simply dropping the DLL over
                    the former version (in the PATH or in $VIMRUNTIME). I expect, though,
                    that upgrading Vim (see above) will be more important than upgrading iconv.

                    Libiconv 1.12 is distributed in source form at
                    http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.12.tar.gz but I suppose
                    these sources are meant primarily for Unix -- I don't know what
                    adaptations (if any) might be necessary to build a Windows DLL from them.


                    Best regards,
                    Tony.
                    --
                    The human mind ordinarily operates at only ten percent of its capacity
                    -- the rest is overhead for the operating system.

                    --~--~---------~--~----~------------~-------~--~----~
                    You received this message from the "vim_multibyte" maillist.
                    For more information, visit http://www.vim.org/maillist.php
                    -~----------~----~----~----~------~----~------~--~---
                  • T.P.S.Nakagawa
                    Thank you Tony, and excuse me too late reply. ... O.K. I install all GnuWin32 and add path to it s bin. I erase old iconv.dll, result of all test pattern is
                    Message 9 of 25 , May 3, 2008
                    • 0 Attachment
                      Thank you Tony, and excuse me too late reply.


                      2008-05-02 18:46 (JST) , Tony Mechelynck sent follow message:

                      >> Someone compile libiconv ver 1.11 or later for Win 32 platform?
                      >> ( I tried 6 month ago, and I fail and busy )
                      >>
                      >
                      > Libiconv 1.9.2 is available precompiled from GnuWin32 from the page

                      O.K. I install all GnuWin32 and add path to it's bin.
                      I erase old iconv.dll, result of all test pattern is same to old iconv.dll.

                      # And Thank you webmaster of http://www.vim.org/ (is Mr. Bram? ) to link change.


                      > This is 7.1.000. I recommend the updated Vim and gvim compiled by Steve
                      > Hall (currently 7.1.293), obtainable from
                      > https://sourceforge.net/project/showfiles.php?group_id=43866&package_id=39721

                      Next, I upgrade vim to 7.1.293.
                      result of all test pattern is same :-(


                      I think, in this time, return to need install new version of iconv.dll for
                      Windows.

                      My Win box is too poor.
                      If I try, need cross compile on Free BSD box. But I didn't try cross compile, too.


                      Best regards,
                      Nakagawa, a.k.a. yaemon

                      --
                      Nakagawa Tsuneo mailto:yaemon@...
                      Web site ( Jpanese only ) http://www.kikansha.jp/~yaemon/

                      --~--~---------~--~----~------------~-------~--~----~
                      You received this message from the "vim_multibyte" maillist.
                      For more information, visit http://www.vim.org/maillist.php
                      -~----------~----~----~----~------~----~------~--~---
                    • Tony Mechelynck
                      ... No problem. [...] ... Make sure to have iconv.dll (and possibly all GnuWin32 executables) in the PATH (which, on Windows, is a semicolon-separated list).
                      Message 10 of 25 , May 3, 2008
                      • 0 Attachment
                        On 03/05/08 20:30, T.P.S.Nakagawa wrote:
                        > Thank you Tony, and excuse me too late reply.

                        No problem.

                        [...]
                        > > Libiconv 1.9.2 is available precompiled from GnuWin32 from the page
                        >
                        > O.K. I install all GnuWin32 and add path to it's bin.
                        > I erase old iconv.dll, result of all test pattern is same to old iconv.dll.

                        Make sure to have iconv.dll (and possibly all GnuWin32 executables) in
                        the PATH (which, on Windows, is a semicolon-separated list). How to set
                        that depends on your Windows version. IIRC, in XP it is at "Control
                        Panel -> System -> Advanced -> Environment Variables" or something similar.

                        >
                        > # And Thank you webmaster of http://www.vim.org/ (is Mr. Bram? ) to link change.
                        [...]

                        I think it is Bram, yes.

                        Best regards,
                        Tony.
                        --
                        "I am ready to meet my Maker. Whether my Maker is prepared for the
                        great ordeal of meeting me is another matter."
                        -- Winston Churchill

                        --~--~---------~--~----~------------~-------~--~----~
                        You received this message from the "vim_multibyte" maillist.
                        For more information, visit http://www.vim.org/maillist.php
                        -~----------~----~----~----~------~----~------~--~---
                      • T.P.S.Nakagawa
                        Thank you Tony. ... Oh , Yes. ( My english is too poor, but not begenner of PC ) I see :fileencodings on gvim , and my _vimrc set ...
                        Message 11 of 25 , May 3, 2008
                        • 0 Attachment
                          Thank you Tony.


                          2008-05-04 8:08 (JST) , Tony Mechelynck sent follow message:
                          >> > Libiconv 1.9.2 is available precompiled from GnuWin32 from the page
                          >>
                          >> O.K. I install all GnuWin32 and add path to it's bin.
                          >> I erase old iconv.dll, result of all test pattern is same to old iconv.dll.
                          >
                          > Make sure to have iconv.dll (and possibly all GnuWin32 executables) in
                          > the PATH (which, on Windows, is a semicolon-separated list). How to set
                          > that depends on your Windows version. IIRC, in XP it is at "Control
                          > Panel -> System -> Advanced -> Environment Variables" or something similar.

                          Oh , Yes. ( My english is too poor, but not begenner of PC )

                          I see :fileencodings on gvim , and my _vimrc set

                          > if has('iconv')
                          > set
                          fileencodings=ascii,iso-2022-jp,utf-8,euc-jp,utf-16,cp932,java,ucs-2-internal,euc-jis0213,utf-16,ISO-8859-1
                          > endif

                          if not path success, `:fileencodings?` will return another value, is'not it?


                          best regards,
                          Nakagawa, a.k.a yaemon


                          --
                          NAKAGAWA Tsuneo mailto:yaemon@...
                          Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                          --~--~---------~--~----~------------~-------~--~----~
                          You received this message from the "vim_multibyte" maillist.
                          For more information, visit http://www.vim.org/maillist.php
                          -~----------~----~----~----~------~----~------~--~---
                        • Tony Mechelynck
                          ... Yes, a Vim build compiled with +iconv/dyn will act as -iconv (has( iconv ) == 0) if it cannot establish contact with _any_ iconv.dll so the other :if
                          Message 12 of 25 , May 3, 2008
                          • 0 Attachment
                            On 04/05/08 03:06, T.P.S.Nakagawa wrote:
                            > Thank you Tony.
                            >
                            >
                            > 2008-05-04 8:08 (JST) , Tony Mechelynck sent follow message:
                            > >> > Libiconv 1.9.2 is available precompiled from GnuWin32 from the page
                            > >>
                            > >> O.K. I install all GnuWin32 and add path to it's bin.
                            > >> I erase old iconv.dll, result of all test pattern is same to old iconv.dll.
                            > >
                            > > Make sure to have iconv.dll (and possibly all GnuWin32 executables) in
                            > > the PATH (which, on Windows, is a semicolon-separated list). How to set
                            > > that depends on your Windows version. IIRC, in XP it is at "Control
                            > > Panel -> System -> Advanced -> Environment Variables" or something similar.
                            >
                            > Oh , Yes. ( My english is too poor, but not begenner of PC )
                            >
                            > I see :fileencodings on gvim , and my _vimrc set
                            >
                            > > if has('iconv')
                            > > set
                            > fileencodings=ascii,iso-2022-jp,utf-8,euc-jp,utf-16,cp932,java,ucs-2-internal,euc-jis0213,utf-16,ISO-8859-1
                            > > endif
                            >
                            > if not path success, `:fileencodings?` will return another value, is'not it?
                            >
                            >
                            > best regards,
                            > Nakagawa, a.k.a yaemon
                            >
                            >

                            Yes, a Vim build compiled with +iconv/dyn will act as -iconv
                            (has("iconv") == 0) if it cannot establish contact with _any_ iconv.dll
                            so the other ":if" branch will be followed, and ":set fileencodings?"
                            (not ":fileencodings" which returns an error) will display a different
                            value.

                            That 'fileencodings' setting makes me wonder:
                            - Shouldn't it start with "ucs-bom" to detect those Unicode files which
                            have a BOM?
                            - Can "ascii" give a fail signal (doesn't Vim treat it as an alias for
                            "latin1")? -- If it can, then it's OK there, but otherwise not.
                            - What is "ucs-2-internal"? Won't it be detected as "utf-16" (which is a
                            superset of UCS-2) by the "utf-16" entry three steps earlier?
                            - Why is "utf-16" mentioned twice? (The second entry does no harm, but
                            will never be used.)
                            - Won't you ever receive UCS-2/UTF-16 files in little-endian ordering
                            (which is standard on Intel ix86 and therefore on Windows)?


                            Best regards,
                            Tony.
                            --
                            If everybody minded their own business, the world would go
                            around a deal faster.
                            -- The Duchess, "Through the Looking Glass"

                            --~--~---------~--~----~------------~-------~--~----~
                            You received this message from the "vim_multibyte" maillist.
                            For more information, visit http://www.vim.org/maillist.php
                            -~----------~----~----~----~------~----~------~--~---
                          • T.P.S.Nakagawa
                            Sorry every typo and send my dirty setting by historical reason. ... Thank you. I correct this sort. ...
                            Message 13 of 25 , May 3, 2008
                            • 0 Attachment
                              Sorry every typo and send my dirty setting by historical reason.


                              2008-05-04 11:24 (JST) , Tony Mechelynck sent follow message:
                              > That 'fileencodings' setting makes me wonder:
                              > - Shouldn't it start with "ucs-bom" to detect those Unicode files which
                              > have a BOM?
                              > - Can "ascii" give a fail signal (doesn't Vim treat it as an alias for
                              > "latin1")? -- If it can, then it's OK there, but otherwise not.
                              > - What is "ucs-2-internal"? Won't it be detected as "utf-16" (which is a
                              > superset of UCS-2) by the "utf-16" entry three steps earlier?
                              > - Why is "utf-16" mentioned twice? (The second entry does no harm, but
                              > will never be used.)
                              > - Won't you ever receive UCS-2/UTF-16 files in little-endian ordering
                              > (which is standard on Intel ix86 and therefore on Windows)?

                              Thank you. I correct this sort.

                              > set
                              fileencodings=ascii,iso-2022-jp,utf-8,euc-jp,java,utf-16,ucs-bomb,cp932,utf-16LE,euc-jis0213,ISO-8859-1


                              First is "ascii" , that's my intention.
                              ( Is it really alias of latin1? latin1 have another code of ascii ).

                              If not exist this first, all ascii code ( ex. programming source ) detected
                              iso-2022-jp and Can't add multibyte comment by UTF-8.

                              I have a trick in this.


                              ----- last of vimrc -------
                              if has( 'autocmd' )
                              source $HOME/.vim/mine/filetype.vim " about filetype
                              source $HOME/.vim/mine/encode.vim " about encoding to save
                              endif

                              ------ cat encode.vim -----
                              " $Id: encode.vim,v 1.6 2007/11/08 05:03:18 yaemon Exp $
                              " set fileencode to save
                              "

                              :autocmd BufNewFile,BufRead *.jis set fileencoding=iso-2022-jp
                              :autocmd BufNewFile,BufRead *.sjis set fileencoding=shift-jis
                              :autocmd BufNewFile,BufRead *.euc-jp set fileencoding=euc-jp
                              :autocmd BufNewFile,BufRead *.mozex set fileencoding=utf-8
                              :autocmd BufNewFile,BufRead *.elm set fileencoding=utf-8
                              :autocmd BufNewFile,BufRead cddbread.* set fileencoding=utf-8

                              :autocmd BufNewFile,BufRead * call s:DefaultSaveCode()

                              function s:DefaultSaveCode()
                              if ( ( &fileencoding == "" ) || ( &fileencoding == "ascii" ) )
                              let &fileencoding = "utf-8"
                              endif
                              endfunction
                              ----------------------------------


                              Isn't is elegant for add UTF-8 to ascii file?



                              Best, Best, regard
                              yaemon

                              --
                              NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                              Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                              --~--~---------~--~----~------------~-------~--~----~
                              You received this message from the "vim_multibyte" maillist.
                              For more information, visit http://www.vim.org/maillist.php
                              -~----------~----~----~----~------~----~------~--~---
                            • T.P.S.Nakagawa
                              Good morning ( it s 8:25 in Japan ) ... Today, I success compile iconv-1.12 for Windows. by cross compile on unix box, mingw32. If you try this, please get
                              Message 14 of 25 , May 4, 2008
                              • 0 Attachment
                                Good morning ( it's 8:25 in Japan )


                                2008-05-04 3:30 (JST) , I sent follow message:
                                > I think, in this time, return to need install new version of iconv.dll for
                                > Windows.
                                >
                                > My Win box is too poor.
                                > If I try, need cross compile on Free BSD box. But I didn't try cross compile, too.

                                Today, I success compile iconv-1.12 for Windows.
                                by cross compile on unix box, mingw32.

                                If you try this, please get here
                                http://www.kikansha.jp/~yaemon/mingw/libiconv-1.12-mingw.zip


                                ...but, I can't correctry open file edited and saved UTF-8 by notepad :(


                                Best regard

                                --
                                NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                                Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                                --~--~---------~--~----~------------~-------~--~----~
                                You received this message from the "vim_multibyte" maillist.
                                For more information, visit http://www.vim.org/maillist.php
                                -~----------~----~----~----~------~----~------~--~---
                              • T.P.S.Nakagawa
                                Hello ... ... I see! On Windows XP, file encoding detect success or not, gvim can t display UTF8 + BOM, if &fileencodings setted. lush-up way is that. ...
                                Message 15 of 25 , May 5, 2008
                                • 0 Attachment
                                  Hello


                                  2008-05-05 8:25 (JST) , I sent follow message:
                                  > Today, I success compile iconv-1.12 for Windows.
                                  <>
                                  > ...but, I can't correctry open file edited and saved UTF-8 by notepad :(

                                  I see!

                                  On Windows XP, file encoding detect success or not, gvim can't display
                                  UTF8 + BOM, if &fileencodings setted.


                                  lush-up way is that.

                                  :set binary
                                  2x
                                  :set fenc=
                                  :w
                                  :set nobinary
                                  :e!


                                  cursed notepad!
                                  but isn't it bug of gvim?


                                  regard
                                  --

                                  NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                                  Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                                  --~--~---------~--~----~------------~-------~--~----~
                                  You received this message from the "vim_multibyte" maillist.
                                  For more information, visit http://www.vim.org/maillist.php
                                  -~----------~----~----~----~------~----~------~--~---
                                • Tony Mechelynck
                                  ... If what you said above is exact, it s a Notepad bug: a UTF-8 BOM is three bytes, a UTF-16 BOM (also used for UCS-2) is two bytes, a UTF-32 BOM is four
                                  Message 16 of 25 , May 5, 2008
                                  • 0 Attachment
                                    On 05/05/08 09:17, T.P.S.Nakagawa wrote:
                                    > Hello
                                    >
                                    >
                                    > 2008-05-05 8:25 (JST) , I sent follow message:
                                    > > Today, I success compile iconv-1.12 for Windows.
                                    > <>
                                    > > ...but, I can't correctry open file edited and saved UTF-8 by notepad :(
                                    >
                                    > I see!
                                    >
                                    > On Windows XP, file encoding detect success or not, gvim can't display
                                    > UTF8 + BOM, if&fileencodings setted.
                                    >
                                    >
                                    > lush-up way is that.
                                    >
                                    > :set binary
                                    > 2x
                                    > :set fenc=
                                    > :w
                                    > :set nobinary
                                    > :e!
                                    >
                                    >
                                    > cursed notepad!
                                    > but isn't it bug of gvim?
                                    >
                                    >
                                    > regard

                                    If what you said above is exact, it's a Notepad bug: a UTF-8 BOM is
                                    three bytes, a UTF-16 BOM (also used for UCS-2) is two bytes, a UTF-32
                                    BOM is four bytes. If there was a two-byte BOM on a UTF-8 file, it's a
                                    bug in the program which produced that file.

                                    When a Windows program (such as WordPad) saves a file as "Unicode text",
                                    it's usually UTF-16le with BOM, which means that the first two bytes are
                                    FF FE and that after that, even bytes are often null bytes.

                                    Best regards,
                                    Tony.
                                    --
                                    Don't take life too seriously -- you'll never get out of it alive.

                                    --~--~---------~--~----~------------~-------~--~----~
                                    You received this message from the "vim_multibyte" maillist.
                                    For more information, visit http://www.vim.org/maillist.php
                                    -~----------~----~----~----~------~----~------~--~---
                                  • T.P.S.Nakagawa
                                    Sorry, Tony. But I pleasure of report next thing of this problem. ... Oh yes. I delete 2 bytes , that displayed in unix UTF-8 console. But by shown od -xc
                                    Message 17 of 25 , May 5, 2008
                                    • 0 Attachment
                                      Sorry, Tony.

                                      But I pleasure of report next thing of this problem.

                                      2008-05-05 23:48 (JST) , Tony Mechelynck sent follow message:
                                      > If what you said above is exact, it's a Notepad bug: a UTF-8 BOM is
                                      > three bytes, a UTF-16 BOM (also used for UCS-2) is two bytes, a UTF-32

                                      Oh yes. I delete 2 bytes , that displayed in unix UTF-8 console.
                                      But by shown "od -xc" command, notepad attach 3 bytes of BOM. sorry.


                                      Then, I report more deep for this problem.
                                      Vim read UTF-8 + BOM , if fileencodings setted, allways display by UTF-8.
                                      so Windows Japanese version ( must display cp932 )
                                      so unix console setted ja_JP.eucJP.

                                      That's all of reason , bad display.

                                      I read 1 hour sources, around *p_fencs setting, but I sleeped.
                                      It's hard of read part of big source.


                                      Best regard, by yaemon.


                                      P.S. now, download page of libiconv is
                                      http://www.kikansha.jp/~yaemon/misc/libiconv
                                      --
                                      NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                                      Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                                      --~--~---------~--~----~------------~-------~--~----~
                                      You received this message from the "vim_multibyte" maillist.
                                      For more information, visit http://www.vim.org/maillist.php
                                      -~----------~----~----~----~------~----~------~--~---
                                    • Tony Mechelynck
                                      ... If your fileencodings starts with ucs-bom , Vim ought to detect correctly any Unicode encoding when there is a BOM without interfering with the
                                      Message 18 of 25 , May 5, 2008
                                      • 0 Attachment
                                        On 06/05/08 04:58, T.P.S.Nakagawa wrote:
                                        > Sorry, Tony.
                                        >
                                        > But I pleasure of report next thing of this problem.
                                        >
                                        > 2008-05-05 23:48 (JST) , Tony Mechelynck sent follow message:
                                        > > If what you said above is exact, it's a Notepad bug: a UTF-8 BOM is
                                        > > three bytes, a UTF-16 BOM (also used for UCS-2) is two bytes, a UTF-32
                                        >
                                        > Oh yes. I delete 2 bytes , that displayed in unix UTF-8 console.
                                        > But by shown "od -xc" command, notepad attach 3 bytes of BOM. sorry.
                                        >
                                        >
                                        > Then, I report more deep for this problem.
                                        > Vim read UTF-8 + BOM , if fileencodings setted, allways display by UTF-8.
                                        > so Windows Japanese version ( must display cp932 )
                                        > so unix console setted ja_JP.eucJP.

                                        If your 'fileencodings' starts with "ucs-bom", Vim ought to detect
                                        correctly any Unicode encoding when there is a BOM without interfering
                                        with the detection of other encodings, unless they may start with one or
                                        more of the following codes and contain not a single invalid byte (or
                                        invalid sequence of bytes) for the corresponding Unicode encoding (I
                                        know that many combinations of bytes higher than 0x7F are invalid in
                                        UTF-8; I'm less sure about the other):

                                        EF BB BF UTF-8
                                        FE FF UTF-16be
                                        FF FE UTF-16le
                                        00 00 FE FF UTF-32be
                                        FF FE 00 00 UTF-32le

                                        Notice that Vim (and any other program with BOM detection) may "guess
                                        wrong" if a file in UTF-16le with BOM starts with a NULL; but I suppose
                                        that such a case is so rare it may be safely ignored.

                                        Notes:
                                        - Even if editing cp932 files, you may set 'encoding' to utf-8
                                        - In GUI mode, anything that 'encoding' can represent, can be displayed
                                        if your 'guifont' has a glyph for it. Characters for which your
                                        'guifont' has no glyph may be represented by a "placeholder" question
                                        mark or hollow box etc.; but if you use the GTK2 GUI (X11 only, thus not
                                        on Windows) it may, in some cases, be clever enough to find an
                                        appropriate glyph in a different font.
                                        - Even if your terminal display is set to accept cp932 output, you may
                                        still set 'encoding' to utf-8 in Console mode if 'termencoding' is set
                                        to cp932, but of course in that case if you edit Unicode (or other
                                        non-cp932) files containing characters which cannot be represented in
                                        cp932, you will get a "placeholder" display (possibly a question mark or
                                        a hollow box) at that position even though the actual contents of the
                                        file are correct.
                                        - The above applies also, of course, with "cp932" replaced everywhere by
                                        "euc-jp".

                                        >
                                        > That's all of reason , bad display.
                                        >
                                        > I read 1 hour sources, around *p_fencs setting, but I sleeped.
                                        > It's hard of read part of big source.

                                        Yes, especially when you're lacking sleep. ;-)

                                        >
                                        >
                                        > Best regard, by yaemon.
                                        >
                                        >
                                        > P.S. now, download page of libiconv is
                                        > http://www.kikansha.jp/~yaemon/misc/libiconv
                                        > --
                                        > NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                                        > Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                                        Best regards,
                                        Tony.
                                        --
                                        "The Good Ship Enterprise" (to the tune of "The Good Ship Lollipop")

                                        On the good ship Enterprise
                                        Every week there's a new surprise
                                        Where the Romulans lurk
                                        And the Klingons often go berserk.

                                        Yes, the good ship Enterprise
                                        There's excitement anywhere it flies
                                        Where Tribbles play
                                        And Nurse Chapel never gets her way.

                                        See Captain Kirk standing on the bridge,
                                        Mr. Spock is at his side.
                                        The weekly menace, ooh-ooh
                                        It gets fried, scattered far and wide.

                                        It's the good ship Enterprise
                                        Heading out where danger lies
                                        And you live in dread
                                        If you're wearing a shirt that's red.
                                        -- Doris Robin and Karen Trimble of The L.A. Filkharmonics

                                        --~--~---------~--~----~------------~-------~--~----~
                                        You received this message from the "vim_multibyte" maillist.
                                        For more information, visit http://www.vim.org/maillist.php
                                        -~----------~----~----~----~------~----~------~--~---
                                      • T.P.S.Nakagawa
                                        Thanks all. ... Oh!? Did I forget write? After libiconv version up to 1.11 , Vim detect all time correct charset. ( If I confidence set fileencoding? ) And,
                                        Message 19 of 25 , May 7, 2008
                                        • 0 Attachment
                                          Thanks all.


                                          2008-05-06 13:30 (JST) , Tony Mechelynck sent follow message:
                                          > If your 'fileencodings' starts with "ucs-bom", Vim ought to detect
                                          > correctly any Unicode encoding when there is a BOM without interfering
                                          > with the detection of other encodings, unless they may start with one or
                                          > more of the following codes and contain not a single invalid byte (or
                                          > invalid sequence of bytes) for the corresponding Unicode encoding (I
                                          > know that many combinations of bytes higher than 0x7F are invalid in
                                          > UTF-8; I'm less sure about the other):

                                          Oh!? Did I forget write?
                                          After libiconv version up to 1.11 , Vim detect all time correct charset.
                                          ( If I confidence "set fileencoding?" )

                                          And, if I wrote in _vimrc "set fileencodings=" ( set null string ),
                                          or started by ucs-bomb,
                                          or overwrite by "set fileencodings=ucs-bomb" ( that's default of Win32),
                                          at all case gvim try diaplay by UTF-8.

                                          Is it only Windows Japanese version's ?
                                          By latin-1 charset, if you wrote (c) _copyright by one character_ or accent + e
                                          by notepad and save by UTF-8, gvim display as you want?


                                          Best regard, by yaemon

                                          --
                                          NAKAGAWA Tsuneo (a.k.a. yaemon ) mailto:yaemon@...
                                          Web site ( Japanese ony ) http://www.kikansha.jp/~yaemon/

                                          --~--~---------~--~----~------------~-------~--~----~
                                          You received this message from the "vim_multibyte" maillist.
                                          For more information, visit http://www.vim.org/maillist.php
                                          -~----------~----~----~----~------~----~------~--~---
                                        Your message has been successfully submitted and would be delivered to recipients shortly.