Loading ...
Sorry, an error occurred while loading the content.

Mac Terminal.app, utf-8 and vim

Expand Messages
  • Peter Vohmann
    Hi folks, I use vim most often remotely (ssh) from my PowerMac G5, and I started to scratch my head why I fail to use german umlaute while everything else
    Message 1 of 10 , Dec 2, 2005
    • 0 Attachment
      Hi folks,

      I use vim most often remotely (ssh) from my PowerMac G5, and I
      started to scratch my head why I fail to use german umlaute while
      everything else seems to be setup and behave propertly.

      When I type "ä" into vim on the mac, I get "ä" instead. You've heard
      it before, right? I have everything set to utf-8, and displaying such
      files is not a problem to vim, it shows them proper as does "cat". I
      prefer utf-8 generally over latin1, or the proprietary macroman (ugh).

      I like to use Terminal.app and compared it to xterm on the Mac.

      I verified that my keyboard submits utf-8 in each case by typing a
      few characters into hexdump, like this. This is the same on every
      terminal:

      $ hexdump -C
      ääääääää <CR><EOF>
      00000000 c3 a4 c3 a4 c3 a4 c3 a4 c3 a4 c3 a4 c3 a4 c3 a4 |
      ääääääää|
      00000010 0a |.|

      I tried the local vim (/usr/bin/vim 6.2) of Mac OS X (10.4.3), the
      mac vim 6.4,
      and a remote vim 6.3.81(ssh to Linux Redhat).
      In all cases I have LC_CTYPE=en_US.utf8 in the shell, and vim derives
      correctly encoding=utf8. Setting termencoding doesn't help.

      mac-vim is ok in gui mode, and is faulty inside Terminal.app as well.

      In any combination, the kind of Terminal matters: vim in Terminal.app
      always translates the key press to "ä". If I start up vim in xterm
      in X11, typing "ä" gives me an "ä"!

      Terminal.app sets TERM=xterm, but there must be an odd difference
      between the two.
      Harr, it happens on input input input!


      And since I ssh to foreign systems, a replacement to vim would not
      exactly be helpful... I'd be thankful for any light shed on this, and
      a solution in the best case...


      Kind regards,
      Peter Vohmann
    • John Love-Jensen
      Hi Peter, In your Terminal / Window Settings... / Terminal Inspector / Display, what does your prefs say for the Character Set Encoding? --Eljay
      Message 2 of 10 , Dec 2, 2005
      • 0 Attachment
        Hi Peter,

        In your Terminal / Window Settings... / Terminal Inspector / Display, what
        does your prefs say for the Character Set Encoding?

        --Eljay
      • Peter Vohmann
        Hi John, You wrote ... Just a second (switching to English...) Character Set Encoding: Unicode (UTF-8) Three cheap shots are free, but sorry, still two to go.
        Message 3 of 10 , Dec 2, 2005
        • 0 Attachment
          Hi John,

          You wrote

          > In your Terminal / Window Settings... / Terminal Inspector /
          > Display, what
          > does your prefs say for the Character Set Encoding?

          Just a second (switching to English...)

          Character Set Encoding:
          Unicode (UTF-8)

          Three cheap shots are free, but sorry, still two to go. ;-)

          Are you able to use national characters in vim inside Terminal.app?

          Peter
        • Jussi Hagman
          ... Hello Peter and welcome to the list. ... Actually I had not but I could reproduce after a little tinkering. You should setup the Terminal.app so that it
          Message 4 of 10 , Dec 2, 2005
          • 0 Attachment
            On 2.12.2005, at 20:53, Peter Vohmann wrote:

            > Hi folks,

            Hello Peter and welcome to the list.

            > When I type "ä" into vim on the mac, I get "ä" instead. You've
            > heard it before, right?

            Actually I had not but I could reproduce after a little tinkering.

            You should setup the Terminal.app so that it would not escape non-
            ascii characters. (uncheck terminal -> window settings -> emulation -
            > escape non ascii chars and save that as defaults)

            Greetings,
            Jussi


            --
            Jussi Hagman, jhagman@..., iChat/AIM: jussihagman, ICQ: 54004113
            Studentbyn 4 D 33, 20540 Åbo, Finland +358 50 56 51 170
          • Peter Vohmann
            ... I found it on the mail archive of this very list. ;-) ... That s it. THANK YOU! Now wouldn t it be nice if this very uncheck was the default of Mac OS X?
            Message 5 of 10 , Dec 2, 2005
            • 0 Attachment
              Jussi Hagman wrote:

              >> When I type "ä" into vim on the mac, I get "ä" instead. You've
              >> heard it before, right?
              >
              > Actually I had not but I could reproduce after a little tinkering.

              I found it on the mail archive of this very list. ;-)

              > You should setup the Terminal.app so that it would not escape non-
              > ascii characters. (uncheck terminal -> window settings -> emulation
              > -> escape non ascii chars and save that as defaults)

              That's it. THANK YOU!

              Now wouldn't it be nice if this very uncheck was the default of Mac
              OS X? It's on by default!!
              The setting is only be relevant to 7 bit connections or some such,
              luckily I don't such iron teletypers anymore. :-)

              Indeed, I fired up ethereal and found Terminal to transmit "\026x",
              where x are the lower 7 bit for each byte, and \026 is also "SYN",
              $16 or ^V. AHH!

              One might also argue that vim could strip the escaping before
              processing the characters,
              but since vi exposes such a nice piece of UNIX history hard wired
              into it's user interface...(!)

              So thanks once more, Jussi, I'm a happy camper again.

              Peter
            • Benji Fisher
              ... Now that you have the answer, if you write it up as a Q&A, I will add it to the list at http://macvim.org/OSX/index.php#FAQ . --Benji Fisher
              Message 6 of 10 , Dec 3, 2005
              • 0 Attachment
                On Fri, Dec 02, 2005 at 09:20:50PM +0100, Peter Vohmann wrote:
                > Jussi Hagman wrote:
                >
                > >>When I type "ä" into vim on the mac, I get "ä" instead. You've
                > >>heard it before, right?
                > >
                > >Actually I had not but I could reproduce after a little tinkering.
                >
                > I found it on the mail archive of this very list. ;-)
                >
                > >You should setup the Terminal.app so that it would not escape non-
                > >ascii characters. (uncheck terminal -> window settings -> emulation
                > >-> escape non ascii chars and save that as defaults)
                >
                > That's it. THANK YOU!

                Now that you have the answer, if you write it up as a Q&A, I will
                add it to the list at http://macvim.org/OSX/index.php#FAQ .

                --Benji Fisher
              • Baptiste Calmès
                ... I have a (maybe) related question. Whether I set the terminal to escape non ascii characters or not, I get the following weird behaviour: In normal mode
                Message 7 of 10 , Dec 3, 2005
                • 0 Attachment
                  Le 3 déc. 05 à 09:38, Benji Fisher a écrit :

                  > On Fri, Dec 02, 2005 at 09:20:50PM +0100, Peter Vohmann wrote:
                  >> Jussi Hagman wrote:
                  >>
                  >>>> When I type "ä" into vim on the mac, I get "ä" instead. You've
                  >>>> heard it before, right?
                  >>>
                  >>> Actually I had not but I could reproduce after a little tinkering.
                  >>
                  >> I found it on the mail archive of this very list. ;-)
                  >>
                  >>> You should setup the Terminal.app so that it would not escape non-
                  >>> ascii characters. (uncheck terminal -> window settings -> emulation
                  >>> -> escape non ascii chars and save that as defaults)
                  >>
                  >> That's it. THANK YOU!

                  I have a (maybe) related question.

                  Whether I set the terminal to escape non ascii characters or not, I
                  get the following weird behaviour:

                  In normal mode (not insert), when I type "^" (to go to the beginning
                  of the line), the character "^" actually appears where the cursor is.

                  If I hit the spacebar, then it disappears and the cursor goes to the
                  beginning of the line (as expected).

                  If, instead, I hit "^" a second time, the character doesn't
                  dissappear, and the cursor goes to the beginning of the line,
                  repacing the first character of the line by "^" (so I actually see
                  two "^" on my line at this stage).
                  Then, if I hit any key, the character "^" at the beginning of the
                  line disappears, but not the one that had appeared first elsewhere in
                  the line.

                  I don't understand what is actually happening. If I use vim in a
                  terminal under X11, that doesn't happen. I must also add that I have
                  a french keyboard and that I use the vim 6.2 coming with tiger, in
                  case it is relevant.

                  Baptiste Calmès
                • Jussi Hagman
                  On 3.12.2005, at 23:20, Baptiste Calmès wrote: Salut, Baptiste! ... In french keymap ^ , l accent circumflex, is not a real character but a soft or
                  Message 8 of 10 , Dec 3, 2005
                  • 0 Attachment
                    On 3.12.2005, at 23:20, Baptiste Calmès wrote:

                    Salut, Baptiste!

                    > In normal mode (not insert), when I type "^" (to go to the
                    > beginning of the line), the character "^" actually appears where
                    > the cursor is.

                    In french keymap "^", l'accent circumflex, is not a real character
                    but a "soft" or "silent" key[1]. Therefore when pressing it no
                    character is actually created but an intermediate underlined "^"
                    shown to provide the user with some feedback. As far as I know this
                    is done by Terminal.app, not vim or another program running. If a
                    suitable character, for instance an "a", is given immediately after
                    this a "â" is produced and sent to vim.

                    > If, instead, I hit "^" a second time, the character doesn't
                    > dissappear, and the cursor goes to the beginning of the line,
                    > repacing the first character of the line by "^" (so I actually see
                    > two "^" on my line at this stage).
                    > Then, if I hit any key, the character "^" at the beginning of the
                    > line disappears, but not the one that had appeared first elsewhere
                    > in the line.

                    This is quite interesting phenomenon, but I'd say this is a bug in
                    Terminal.app, not in vim. If you press ctrl-l to force a redraw you
                    see that the line has the expected characters. Terminal.app just
                    seems to loose it if cursor is moved when it is not expecting it.

                    I tried this also with iTerm, which is another terminal for OS X. It
                    seems to handle the silent characters differently, in a less buggy
                    way. I guess we could work around this bug by redrawing the line when
                    "^" command is issued but I'd be very hesitant to see an OS X
                    specific fix in vim to work around a Terminal.app bug.

                    Disclaimer: I wrote this without a deeper analyze of the problem so I
                    might have missed something important..

                    [1] There must be a real name for that, which I unfortunately don't
                    know.


                    --
                    Jussi Hagman, jhagman@..., iChat/AIM: jussihagman, ICQ: 54004113
                    Studentbyn 4 D 33, 20540 Åbo, Finland +358 50 56 51 170
                  • Baptiste Calmès
                    ... Yes, indeed. ... You are right, I hadn t noticed that. ... Yes, this wouldn t be reasonnable. Since apple ships vim with tiger, it might want to improve
                    Message 9 of 10 , Dec 4, 2005
                    • 0 Attachment
                      Le 3 déc. 05 à 19:07, Jussi Hagman a écrit :
                      >
                      > In french keymap "^", l'accent circumflex, is not a real character
                      > but a "soft" or "silent" key[1]. Therefore when pressing it no
                      > character is actually created but an intermediate underlined "^"
                      > shown to provide the user with some feedback. As far as I know this
                      > is done by Terminal.app, not vim or another program running. If a
                      > suitable character, for instance an "a", is given immediately after
                      > this a "â" is produced and sent to vim.

                      Yes, indeed.

                      >> If, instead, I hit "^" a second time, the character doesn't
                      >> dissappear, and the cursor goes to the beginning of the line,
                      >> repacing the first character of the line by "^" (so I actually see
                      >> two "^" on my line at this stage).
                      >> Then, if I hit any key, the character "^" at the beginning of the
                      >> line disappears, but not the one that had appeared first elsewhere
                      >> in the line.
                      >
                      > This is quite interesting phenomenon, but I'd say this is a bug in
                      > Terminal.app, not in vim. If you press ctrl-l to force a redraw you
                      > see that the line has the expected characters.

                      You are right, I hadn't noticed that.

                      > Terminal.app just seems to loose it if cursor is moved when it is
                      > not expecting it.
                      >
                      > I tried this also with iTerm, which is another terminal for OS X.
                      > It seems to handle the silent characters differently, in a less
                      > buggy way. I guess we could work around this bug by redrawing the
                      > line when "^" command is issued but I'd be very hesitant to see an
                      > OS X specific fix in vim to work around a Terminal.app bug.

                      Yes, this wouldn't be reasonnable. Since apple ships vim with tiger,
                      it might want to improve its termina.app, so that vim works correctly
                      with it (using any keymap). Maybe I'll try submitting this to apple
                      as a bug, if I find a place to do it.

                      Thank you for your answer.

                      Baptiste Calmès
                    • Peter Vohmann
                      Hi Benji, Ah yes, I almost missed that request. Of course, here you are: Q: How can I use national characters when running vim in a Terminal window? I get
                      Message 10 of 10 , Dec 6, 2005
                      • 0 Attachment
                        Hi Benji,

                        Ah yes, I almost missed that request. Of course, here you are:


                        Q: How can I use national characters when running vim in a Terminal
                        window? I get funny characters or extra white space!
                        A: Go to Terminal.app -> Menu Terminal -> Window Settings. Select
                        Emulation from the Popup menu, un-check the item Escape non-ASCII
                        characters (terminal sends ctrl-V for quoting). Then select Display
                        from the Popup menu, set Character Set Encoding to Unicode (UTF-8),
                        if desired. Click on "Use settings as Default."

                        At last, vim defaults to "set encoding=latin1". To use Unicode
                        instead, add the following line to your .bashrc, then open a new shell:
                        export LC_CTYPE=en_US.UTF-8


                        Kind regards,
                        Peter


                        Am 03.12.2005 um 17:38 schrieb Benji Fisher:

                        > On Fri, Dec 02, 2005 at 09:20:50PM +0100, Peter Vohmann wrote:
                        >> Jussi Hagman wrote:
                        >>
                        >>>> When I type "ä" into vim on the mac, I get "ä" instead. You've
                        >>>> heard it before, right?
                        >>>
                        >>> Actually I had not but I could reproduce after a little tinkering.
                        >>
                        >> I found it on the mail archive of this very list. ;-)
                        >>
                        >>> You should setup the Terminal.app so that it would not escape non-
                        >>> ascii characters. (uncheck terminal -> window settings -> emulation
                        >>> -> escape non ascii chars and save that as defaults)
                        >>
                        >> That's it. THANK YOU!
                        >
                        > Now that you have the answer, if you write it up as a Q&A, I will
                        > add it to the list at http://macvim.org/OSX/index.php#FAQ .
                        >
                        > --Benji Fisher
                      Your message has been successfully submitted and would be delivered to recipients shortly.