Loading ...
Sorry, an error occurred while loading the content.

Re: Combining characters and case-changing

Expand Messages
  • Tony Mechelynck
    ... The problem I saw existed only in text containing combining characters (i.e. Unicode codepoints which are drawn superimposed on the preceding character,
    Message 1 of 5 , Aug 1, 2010
    • 0 Attachment
      On 01/08/10 13:43, Jakson A. Aquino wrote:
      > On Sun, Aug 1, 2010 at 5:59 AM, Tony Mechelynck
      > <antoine.mechelynck@...> wrote:
      >> Using latest Vim 7.3c, compiled 2010-07-31 21:22:42 +0200 immediately after
      >> pulling.
      >>
      >> Hitting ~ (tilde) on a letter + combining character gives faulty result.
      >>
      >>
      >> Reproducible: Always.
      >>
      >>
      >> Steps to reproduce(1):
      >> 1. Type text in some case-sensitive script (not Hebrew or Arabic) with some
      >> combining characters in it (e.g. Russian with combining acute accent(s)).
      >> 2. In Normal mode, move the cursor over a letter with combining accent and
      >> hit ~ (tilde).
      >>
      >> Expected result:
      >> The letter should change case and keep its combining accent.
      >>
      >> Actual result:
      >> The accented letter is replaced by both its upper- and lower-case variants,
      >> without the accent.
      >>
      >>
      >> Steps to reproduce(2)
      >> 1. Like before, type text with some combining characters in it.
      >> 2. Select a section of text (I used V for single-line linewise-visual).
      >> 3. Hit the tilde.
      >>
      >> Expected result:
      >> The text should change case, with the combining characters remaining where
      >> they belong.
      >>
      >> Actual result:
      >> - Accented characters become doubled, losing their accent.
      >> - At the end of the selection, the last characters (as many as there were
      >> accents) are not case-toggled.
      >>
      >>
      >> Additional info:
      >> I haven't tested what happens with _several_ combining characters on a
      >> single letter (e.g. non-precomposed Classical Greek with breathing, accent
      >> and/or iota-subscript/adscript on the same vowel), or with spacing and
      >> combining characters of different byte-length (Cyrillic letters and
      >> combining-acute are all two bytes per codepoint in UTF-8). Neither did I
      >> check that ~ in case-neutral text with combining characters (such as
      >> vocalised Semitic text) is a no-op.
      >
      > Here it happens what was expected. I tested with tutor.utf-8 (fr, ru,
      > el, vi). I'm using Ubuntu 10.04 in pt_BR.UTF-8 locale with Vim from
      > current hg.
      >
      > Best regards,
      >
      > Jakson
      >

      The problem I saw existed only in text containing combining characters
      (i.e. Unicode codepoints which are drawn superimposed on the preceding
      character, such as e.g. U+0301 COMBINING ACUTE ACCENT). When the accent
      is part of a spacing codepoint (e.g. é U+00E9 LATIN SMALL LETTER E WITH
      ACUTE) there was no problem. With Vim, combining characters have to be
      entered each separately, and if 'delcombine' is on they can also be
      removed one-by-one, separately from the letter over which they are drawn.

      I was using gvim; if you were using Vim in an mlterm console (with
      'termbidi' on) the drawing of the full line (including displaying words
      in RTL scripts right-to-left even in LTR sentences, and displaying
      combining characters superimposed if there are any) is handled by the
      terminal, not by Vim: this might make a difference.

      In Russian, for instance, the combining acute accent is the only way to
      add an accent over a Cyrillic letter; it is used to indicate stress in
      dictionaries, in reading books for children or foreigners learning the
      language, and sometimes to avoid ambiguity as in "у белых теперь бо́льшая
      выгода" "u byelyh teper' BOLshaya vygoda", "White now has a bigger
      advantage", which I saw once in a Russian book about chess; without the
      written accent (у белых теперь большая выгода) it would be read "u
      byelyh teper' bolSHAya vygoda", "White now has a big advantage". In
      SeaMonkey I see this accent over the letter л, which is wrong; Vim
      correctly displays it over the о; but SeaMonkey, unlike Vim, has the
      possibility to place the cursor between the combining character and the
      spacing character preceding it.

      Similarly, Hebrew and Arabic (where case differences don't exist)
      optionally use combining characters to denote short vowels etc. (again,
      in dictionaries, reading books for children or foreigners, sacred
      writings, and in any text to avoid ambiguity); in many scripts the
      possibility exists to use combining characters, but precombined
      characters, when they exist, are often preferred in practice.

      Bram corrected the problem in a changeset dated 14:22:48 +0200 today
      (that would be shortly after your reply), and it works. Thanks Bram!


      Best regards,
      Tony.
      --
      Snoring is prohibited unless all bedroom windows are closed and securely
      locked.
      [real standing law in Massachusetts, United States of America]

      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • Tony Mechelynck
      ... Well, previously (in the Ctrl-R case) I had done it, but the Russian characters in my text quoted in your answer were totally garbled (wrong Content-Type
      Message 2 of 5 , Aug 1, 2010
      • 0 Attachment
        On 01/08/10 16:06, Bram Moolenaar wrote:
        >
        > Tony Mechelynck wrote:
        >
        >> Using latest Vim 7.3c, compiled 2010-07-31 21:22:42 +0200 immediately
        >> after pulling.
        >>
        >> Hitting ~ (tilde) on a letter + combining character gives faulty result.
        >>
        >>
        >> Reproducible: Always.
        >>
        >>
        >> Steps to reproduce(1):
        >> 1. Type text in some case-sensitive script (not Hebrew or Arabic) with
        >> some combining characters in it (e.g. Russian with combining acute
        >> accent(s)).
        >> 2. In Normal mode, move the cursor over a letter with combining accent
        >> and hit ~ (tilde).
        >>
        >> Expected result:
        >> The letter should change case and keep its combining accent.
        >>
        >> Actual result:
        >> The accented letter is replaced by both its upper- and lower-case
        >> variants, without the accent.
        >
        > I'll fix this. Note that for the example it's very useful to give the
        > actual text to reproduce it on.

        Well, previously (in the Ctrl-R case) I had done it, but the Russian
        characters in my text quoted in your answer were totally garbled (wrong
        Content-Type charset maybe?). You could have reused the same example
        text, or any text with any combining character; I used the combining
        acute accent U+0301; there are other combining codepoints in the same
        Unicode block.

        >
        >> Steps to reproduce(2)
        >> 1. Like before, type text with some combining characters in it.
        >> 2. Select a section of text (I used V for single-line linewise-visual).
        >> 3. Hit the tilde.
        >>
        >> Expected result:
        >> The text should change case, with the combining characters remaining
        >> where they belong.
        >>
        >> Actual result:
        >> - Accented characters become doubled, losing their accent.
        >> - At the end of the selection, the last characters (as many as there
        >> were accents) are not case-toggled.
        >
        > I can't reproduce doubling the characters. Do you have 'delcombine' set
        > perhaps?

        Yes I do. This way I can easily correct an error if I notice that I have
        put the accent on the wrong syllable, or the wrong diacritic on the
        right letter.

        >
        >> Additional info:
        >> I haven't tested what happens with _several_ combining characters on a
        >> single letter (e.g. non-precomposed Classical Greek with breathing,
        >> accent and/or iota-subscript/adscript on the same vowel), or with
        >> spacing and combining characters of different byte-length (Cyrillic
        >> letters and combining-acute are all two bytes per codepoint in UTF-8).
        >> Neither did I check that ~ in case-neutral text with combining
        >> characters (such as vocalised Semitic text) is a no-op.
        >
        > I have fixed one problem. Please check if you can still reproduce the
        > others. If so, please include the text.
        >

        No, the tilde now works correctly, both in Normal mode (when toggling
        one character at a time, with 'tildeop' at its default off setting), and
        in linewise Visual mode (when toggling a full line in one operation).
        The length of the Visual text operated upon is now also correct.


        Best regards,
        Tony.
        --
        "It took me fifteen years to discover that I had no talent for writing,
        but I couldn't give up because by that time I was too famous."
        -- Robert Benchly

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      Your message has been successfully submitted and would be delivered to recipients shortly.