Loading ...
Sorry, an error occurred while loading the content.

CarbonVim encoding problem

Expand Messages
  • Sébastien Pierre
    The problem is the following: Carbon Vim does internally handle the encodings well, this means that it can internally properly interpret and convert between
    Message 1 of 14 , May 24, 2002
    • 0 Attachment
      The problem is the following: Carbon Vim does internally handle the
      encodings well, this means that it can internally properly interpret and
      convert between the different available encoding, but it does not do
      encoding translation from keyboard input and to screen output.

      When Vim encoding is set to latin-1, which is a different encoding from
      the MacRoman used by default, a keypress on the 'é' (<8E>) keyboard will
      not be translated to the character associated to 'é' (<E9>) in latin-1,
      but left as <8E> in the latin-1 file.

      This problem would be visible if the display did the conversion from
      latin-1 to MacRoman, in this case it would try to display the <8E>
      character from latin-1 to MacRoman. This is not the case, so it displays
      the <8E> character of Latin-1 as the <8E> character of MacRoman, so that
      we don't notice something went wrong.

      To fix this, the keyboard input must be fixed to properly convert the
      input key code to vim internal keycode, the same thing being necessary
      for the character outputting. I don't think the change would be hard to
      make as this would be just like overloading the default behaviour of
      character reads and writes using the encoding translation tables.

      Hope this is a clear explainantion ;)

      -- Sébastien.

      --
      «My friends says we're like the dinosaurs, only we are doing
      ourselves in much faster than they ever did.»
      <http://www.type-z.org> -- Porno For Pyros, Pets
    • Bram Moolenaar
      ... Perhaps termencoding needs to be set? ... Alternatively, you can set encoding to 8bit-MacRoman . Don t know how well this works though. You probably
      Message 2 of 14 , May 24, 2002
      • 0 Attachment
        Sébastien Pierre wrote:

        > The problem is the following: Carbon Vim does internally handle the
        > encodings well, this means that it can internally properly interpret and
        > convert between the different available encoding, but it does not do
        > encoding translation from keyboard input and to screen output.
        >
        > When Vim encoding is set to latin-1, which is a different encoding from
        > the MacRoman used by default, a keypress on the 'é' (<8E>) keyboard will
        > not be translated to the character associated to 'é' (<E9>) in latin-1,
        > but left as <8E> in the latin-1 file.

        Perhaps 'termencoding' needs to be set?

        > This problem would be visible if the display did the conversion from
        > latin-1 to MacRoman, in this case it would try to display the <8E>
        > character from latin-1 to MacRoman. This is not the case, so it displays
        > the <8E> character of Latin-1 as the <8E> character of MacRoman, so that
        > we don't notice something went wrong.

        Alternatively, you can set 'encoding' to "8bit-MacRoman". Don't know
        how well this works though. You probably also need to set
        'fileencodings' to include "8bit-MacRoman" instead of "latin1".

        > To fix this, the keyboard input must be fixed to properly convert the
        > input key code to vim internal keycode, the same thing being necessary
        > for the character outputting. I don't think the change would be hard to
        > make as this would be just like overloading the default behaviour of
        > character reads and writes using the encoding translation tables.

        This is possible by setting 'termencoding', but conversion for 8-bit
        characters usually means some characters will be dropped. Setting
        'encoding' might work better.

        --
        hundred-and-one symptoms of being an internet addict:
        55. You ask your doctor to implant a gig in your brain.

        /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
        /// Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim \\\
        \\\ Project leader for A-A-P -- http://www.a-a-p.org ///
        \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
      • Sébastien Pierre
        Hi! ... I tried : set termencoding=8bit-MacRoman set encoding=latin-1 set fileencoding=latin-1 set fileencodings=utf-8,latin-1,8bit-MacRoman and set
        Message 3 of 14 , May 25, 2002
        • 0 Attachment
          Hi!

          Le vendredi 24 mai 2002, à 08:41 PM, Bram Moolenaar a écrit :

          > Perhaps 'termencoding' needs to be set?

          I tried :

          set termencoding=8bit-MacRoman
          set encoding=latin-1
          set fileencoding=latin-1
          set fileencodings=utf-8,latin-1,8bit-MacRoman

          and

          set termencoding=latin-1
          set encoding=8bit-MacRoman
          set fileencoding=latin-1
          set fileencodings=utf-8,latin-1,8bit-MacRoman

          and

          set termencoding=8bit-MacRoman
          set encoding=8bit-MacRoman
          set fileencoding=latin-1
          set fileencodings=utf-8,latin-1,8bit-MacRoman

          this last solution, even with an explicit fileencoding=latin-1 did not
          produce the expected <E9> charcater instead of the <8E> when pressing
          'é'.

          > Alternatively, you can set 'encoding' to "8bit-MacRoman". Don't know
          > how well this works though. You probably also need to set
          > 'fileencodings' to include "8bit-MacRoman" instead of "latin1".

          Removing the termencoding in my .vimrc did not change anything, removing
          latin-1 from the filencodings did nothing as well.

          >> To fix this, the keyboard input must be fixed to properly convert the
          >> input key code to vim internal keycode, the same thing being necessary
          >> for the character outputting. I don't think the change would be hard to
          >> make as this would be just like overloading the default behaviour of
          >> character reads and writes using the encoding translation tables.
          >
          > This is possible by setting 'termencoding', but conversion for 8-bit
          > characters usually means some characters will be dropped. Setting
          > 'encoding' might work better.

          If I understood well:

          - termencoding specifies the encoding in which keyboard input characters
          are mapped. For CarbonVim this is 8bit-MacRoman
          - encoding specifies the display encoding, which should also be
          8bit-MacRoman, but as shown before the encoding variable does not seem
          to have any effect, even when used without termencoding
          - fileencoding forces a specific encoding for the current file, trying
          to do the conversion between the encoding. This should be latin-1, as
          latin-1 is more likely to be read properly on other Unices than MacRoman
          - fileencodings lists the different encodings that are tried when
          reading a file, so this should be utf-8 first, which will most likely
          fail, then latin-1 and eventually MacRoman.

          I think the problem may be that CarbonVim is not linked with iconv, so
          that it does not actually do any conversion, but I must also admit that
          I still don't have a crystal clear understanding of the four above
          variables...

          Thanks for your help,

          -- Sébastien.


          --
          «Il vaut mieux suivre une voix stupide que l'on connait,
          plutôt qu'une intelligente que l'on ne connait pas.»
          <http://www.type-z.org> -- No comment on Jacques C.
        • Bram Moolenaar
          ... Putting both latin-1 and 8bit-MacRoman in fileencodings will always result in using latin-1, because there is no way to detect that a file isn t latin-1.
          Message 4 of 14 , May 25, 2002
          • 0 Attachment
            Sébastien Pierre wrote:

            > > Perhaps 'termencoding' needs to be set?
            >
            > I tried :
            >
            > set termencoding=8bit-MacRoman
            > set encoding=latin-1
            > set fileencoding=latin-1
            > set fileencodings=utf-8,latin-1,8bit-MacRoman

            Putting both latin-1 and 8bit-MacRoman in 'fileencodings' will always
            result in using latin-1, because there is no way to detect that a file
            isn't latin-1. Setting 'fileencoding' has no effect, it will be changed
            to one of the entries in 'fileencodings' as soon as you load a file.

            > set termencoding=8bit-MacRoman
            > set encoding=8bit-MacRoman
            > set fileencoding=latin-1
            > set fileencodings=utf-8,latin-1,8bit-MacRoman
            >
            > this last solution, even with an explicit fileencoding=latin-1 did not
            > produce the expected <E9> charcater instead of the <8E> when pressing
            > 'é'.

            When 'termencoding' is equal to 'encoding' or empty no conversion is
            done.

            > > Alternatively, you can set 'encoding' to "8bit-MacRoman". Don't know
            > > how well this works though. You probably also need to set
            > > 'fileencodings' to include "8bit-MacRoman" instead of "latin1".
            >
            > Removing the termencoding in my .vimrc did not change anything, removing
            > latin-1 from the filencodings did nothing as well.

            Did you check the value of 'fileencoding' after loading a file?

            > If I understood well:
            >
            > - termencoding specifies the encoding in which keyboard input characters
            > are mapped. For CarbonVim this is 8bit-MacRoman

            Yes, and for terminal output it also specifies conversion for displayed
            characters.

            > - encoding specifies the display encoding, which should also be
            > 8bit-MacRoman, but as shown before the encoding variable does not seem
            > to have any effect, even when used without termencoding

            'encoding' specifies the internal encoding. For 8-bit encodings the
            name isn't important for the internal stuff, except that it's used for
            conversions. For the GUI there is no conversion for displaying, thus
            'encoding' also sets the encoding for displaying. You need to use the
            right 'guifont' to display characters for 'encoding'.

            > - fileencoding forces a specific encoding for the current file, trying
            > to do the conversion between the encoding. This should be latin-1, as
            > latin-1 is more likely to be read properly on other Unices than MacRoman

            Only for writing. When reading 'fileencoding' is set from the values in
            'termencodings'. Or it's made empty when the file can't be read in any
            of these (which means 'encoding' is used). Unless you use ":edit
            ++enc=name file", then it's forced to use "name".

            > - fileencodings lists the different encodings that are tried when
            > reading a file, so this should be utf-8 first, which will most likely
            > fail, then latin-1 and eventually MacRoman.

            As said above, if you put latin-1 first, MacRoman will never be used,
            because latin-1 can always be used for any file (well, unless conversion
            from latin-1 to 'encoding' fails, which would be very bad).

            > I think the problem may be that CarbonVim is not linked with iconv, so
            > that it does not actually do any conversion, but I must also admit that
            > I still don't have a crystal clear understanding of the four above
            > variables...

            If iconv isn't present, you can see this in the output of ":version".
            And you can test it with the iconv() function.

            --
            hundred-and-one symptoms of being an internet addict:
            57. You begin to wonder how on earth your service provider is allowed to call
            200 hours per month "unlimited."

            /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
            /// Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim \\\
            \\\ Project leader for A-A-P -- http://www.a-a-p.org ///
            \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
          • Sébastien Pierre
            Hi! ... I tried: set termencoding=8bit-MacRoman set encoding=latin-1 set fileencodings=8bit-MacRoman and also tried to remove the termencoding. Both did no
            Message 5 of 14 , May 25, 2002
            • 0 Attachment
              Hi!

              Le samedi 25 mai 2002, à 01:27 PM, Bram Moolenaar a écrit :
              > When 'termencoding' is equal to 'encoding' or empty no conversion is
              > done.

              I tried:

              set termencoding=8bit-MacRoman
              set encoding=latin-1
              set fileencodings=8bit-MacRoman

              and also tried to remove the termencoding. Both did no solve the
              problem: I still have <8E> instead of <E9>.

              >> Removing the termencoding in my .vimrc did not change anything,
              >> removing
              >> latin-1 from the filencodings did nothing as well.
              >
              > Did you check the value of 'fileencoding' after loading a file?

              I can't remember nor find in the manual how to do that (echo just
              doesn't work with this kind of... variable?). I also tested with
              :fileencoding=latin-1 and :e --enc=latin 1 with the above configuration,
              and I still got the <8E>.

              I have checked with :version, and iconv has not been compiled in. This
              may be the reason why the encodings do not work properly?

              Thanks,

              -- Sébastien.

              --
              «Le soleil est Dieu»
              <http://www.type-z.org> -- Turner, avant de mourir
            • Bram Moolenaar
              ... Yes, without iconv no conversion is possible. -- Q: How to decide if I should clean my house or work on improving Vim? A: Depends on which has more bugs.
              Message 6 of 14 , May 25, 2002
              • 0 Attachment
                Sébastien Pierre wrote:

                > I can't remember nor find in the manual how to do that (echo just

                :set fileencoding

                > I have checked with :version, and iconv has not been compiled in. This
                > may be the reason why the encodings do not work properly?

                Yes, without iconv no conversion is possible.

                --
                Q: How to decide if I should clean my house or work on improving Vim?
                A: Depends on which has more bugs.

                /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                /// Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim \\\
                \\\ Project leader for A-A-P -- http://www.a-a-p.org ///
                \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
              • Benji Fisher
                ... Should I add something like Enable font conversions with iconv to the list of bugs on my web page? Or is this something that I can add at compile time?
                Message 7 of 14 , May 25, 2002
                • 0 Attachment
                  On Saturday, May 25, 2002, at 02:17 PM, Bram Moolenaar wrote:

                  >
                  > Sébastien Pierre wrote:
                  >
                  >> I have checked with :version, and iconv has not been compiled in. This
                  >> may be the reason why the encodings do not work properly?
                  >
                  > Yes, without iconv no conversion is possible.

                  Should I add something like "Enable font conversions with iconv" to
                  the list of bugs on my web page? Or is this something that I can add at
                  compile time?

                  > --
                  > Q: How to decide if I should clean my house or work on improving Vim?
                  > A: Depends on which has more bugs.

                  I think you should multiply the number of bugs by the number of
                  people bothered by each bug. ;)

                  --Benji Fisher
                • mc
                  ... ... and divide by a function of the distance of the bothered people from the bugs in the house
                  Message 8 of 14 , May 25, 2002
                  • 0 Attachment
                    >> Q: How to decide if I should clean my house or work on improving Vim?
                    >> A: Depends on which has more bugs.
                    >
                    > I think you should multiply the number of bugs by the number of
                    > people bothered by each bug. ;)


                    ... and divide by a function of the distance of the bothered people from
                    the bugs in the house
                  • Sébastien Pierre
                    ... For people who have Fink (like me), libiconv is available. The problem is that when you add -I/sw/include in CarbonVim makefile there are gcc duplicate
                    Message 9 of 14 , May 26, 2002
                    • 0 Attachment
                      Le dimanche 26 mai 2002, à 04:03 AM, Benji Fisher a écrit :

                      > On Saturday, May 25, 2002, at 02:17 PM, Bram Moolenaar wrote:
                      >
                      >>
                      >> Sébastien Pierre wrote:
                      >>
                      >>> I have checked with :version, and iconv has not been compiled in. This
                      >>> may be the reason why the encodings do not work properly?
                      >>
                      >> Yes, without iconv no conversion is possible.
                      >
                      > Should I add something like "Enable font conversions with iconv"
                      > to the list of bugs on my web page? Or is this something that I can
                      > add at compile time?

                      For people who have Fink (like me), libiconv is available. The problem
                      is that when you add -I/sw/include in CarbonVim makefile there are "gcc
                      duplicate declaration" errors. Iconv is a compilation option, so if you
                      know how to enable the makefile to look for additional libs in /sw/lib
                      and headers in /sw/include, this would be perfect :)

                      Thanks,

                      -- Sébastien.

                      --
                      «Big lies, big times / This love is not what we're about / It's
                      too late, and I'm too straight / It's time to blow this fire
                      out.»
                      <http://www.type-z.org> -- UNKLE, Psyence Fiction - Bloodstain
                    • Bram Moolenaar
                      ... If you do have the iconv library, configure should find it automatically. However, I don t know if this works on the mac. You could try defining
                      Message 10 of 14 , May 26, 2002
                      • 0 Attachment
                        Benji Fisher wrote:

                        > > Sébastien Pierre wrote:
                        > >
                        > >> I have checked with :version, and iconv has not been compiled in. This
                        > >> may be the reason why the encodings do not work properly?
                        > >
                        > > Yes, without iconv no conversion is possible.
                        >
                        > Should I add something like "Enable font conversions with iconv" to
                        > the list of bugs on my web page? Or is this something that I can add at
                        > compile time?

                        If you do have the iconv library, configure should find it
                        automatically. However, I don't know if this works on the mac. You
                        could try defining USE_ICONV and see what happens. Perhaps define
                        HAVE_ICONV_H as well.

                        Another problem is that you might have the iconv library, but someone
                        who runs your compiled binary doesn't. The best way to solve this is by
                        dynamically loading the library. But currently this is only implemented
                        for MS-Windows. Would be a great help to have this on more systems!

                        > > --
                        > > Q: How to decide if I should clean my house or work on improving Vim?
                        > > A: Depends on which has more bugs.
                        >
                        > I think you should multiply the number of bugs by the number of
                        > people bothered by each bug. ;)

                        Then I need someone else to clean my house! :-)

                        --
                        hundred-and-one symptoms of being an internet addict:
                        60. As your car crashes through the guardrail on a mountain road, your first
                        instinct is to search for the "back" button.

                        /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                        /// Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim \\\
                        \\\ Project leader for A-A-P -- http://www.a-a-p.org ///
                        \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                      • MURAOKA Taro
                        ... MacOS X (Carbon) has a set of APIs which called Text Encoding Conversion Manager . It works as like as iconv library. So I have wrote iconv emulation
                        Message 11 of 14 , May 26, 2002
                        • 0 Attachment
                          > Another problem is that you might have the iconv library, but someone
                          > who runs your compiled binary doesn't. The best way to solve this is by
                          > dynamically loading the library. But currently this is only implemented
                          > for MS-Windows. Would be a great help to have this on more systems!

                          MacOS X (Carbon) has a set of APIs which called "Text Encoding
                          Conversion Manager". It works as like as iconv library. So I have
                          wrote iconv emulation layer for MacOS X using these APIs, and now I am
                          testing for Vim. This emulation layer is not so big as iconv library is,
                          so it could be included in vim.

                          Please try below patch and Makefile.
                          http://www.kaoriya.net/testdir/VimMacOSXIconvMakeSet.tar.bz2

                          In this layer, I add some aliases for Japansese encoding name. I'm
                          glad, if anyone add aliases for another languages (ex. Chinese, Korean)

                          Regards.
                          ----
                          MURAOKA Taro <koron@...>
                        Your message has been successfully submitted and would be delivered to recipients shortly.