Loading ...
Sorry, an error occurred while loading the content.

[BUG] Quote characters in expression mapping are replaced with garbage

Expand Messages
  • ZyX
    Consider the following command: vim -u NONE -c noremap! - ‘ -c normal i- -c wq! result.out result.out: $ cat result.out | hexdump -C
    Message 1 of 4 , Jan 4, 2011
    • 0 Attachment
      Consider the following command:

      vim -u NONE -c 'noremap! <expr> - "‘"' \
      -c 'normal i-' \
      -c 'wq! result.out'

      result.out:
      $ cat result.out | hexdump -C
      00000000 c3 a2 c2 80 c2 98 0a |.......|
      00000007

      Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
      Everything works fine if it is not expr mapping or if I replace «"‘"» with
      «"\u2018"». Same issue with opening double english quote (u201C) and second
      level opening russian quote (u201E).
    • Bram Moolenaar
      ... Yes, that is wrong, each byte is used as a character. -- hundred-and-one symptoms of being an internet addict: 84. Books in your bookcase bear the names
      Message 2 of 4 , Jan 4, 2011
      • 0 Attachment
        ZyX wrote:

        > Consider the following command:
        >
        > vim -u NONE -c 'noremap! <expr> - "‘"' \
        > -c 'normal i-' \
        > -c 'wq! result.out'
        >
        > result.out:
        > $ cat result.out | hexdump -C
        > 00000000 c3 a2 c2 80 c2 98 0a |.......|
        > 00000007
        >
        > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
        > Everything works fine if it is not expr mapping or if I replace «"‘"» with
        > «"\u2018"». Same issue with opening double english quote (u201C) and second
        > level opening russian quote (u201E).

        Yes, that is wrong, each byte is used as a character.

        --
        hundred-and-one symptoms of being an internet addict:
        84. Books in your bookcase bear the names Bongo, WinSock and Inside OLE

        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
        /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
        \\\ an exciting new programming language -- http://www.Zimbu.org ///
        \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • Peter Odding
        ... I encountered this bug* while trying to map and to insert smart quotes for my notes.vim plug-in. Fortunately I found a simple workaround (for my use
        Message 3 of 4 , Jan 4, 2011
        • 0 Attachment
          > Consider the following command:
          >
          > vim -u NONE -c 'noremap!<expr> - "‘"' \
          > -c 'normal i-' \
          > -c 'wq! result.out'
          >
          > result.out:
          > $ cat result.out | hexdump -C
          > 00000000 c3 a2 c2 80 c2 98 0a |.......|
          > 00000007
          >
          > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
          > Everything works fine if it is not expr mapping or if I replace «"‘"» with
          > «"\u2018"». Same issue with opening double english quote (u201C) and second
          > level opening russian quote (u201E).

          I encountered this bug* while trying to map ' and " to insert smart
          quotes for my notes.vim plug-in. Fortunately I found a simple workaround
          (for my use case at least): Don't embed the special characters directly
          in the <expr> mapping but call a function that returns the special
          characters instead (in my case the function is defined in a different
          script but I assume this doesn't matter).

          - Peter Odding

          *
          https://github.com/xolox/vim-notes/blob/3e4ec8fe4b2f68cdedc788f7c503308f618d4865/autoload.vim#L396

          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • Yukihiro Nakadaira
          ... See :help :map- and search CSI. This applies to 0x80 byte, too. map.txt: Note that there are some tricks to make special keys work and escape CSI
          Message 4 of 4 , Jan 5, 2011
          • 0 Attachment
            ZyX wrote:
            > Consider the following command:
            >
            > vim -u NONE -c 'noremap!<expr> - "‘"' \
            > -c 'normal i-' \
            > -c 'wq! result.out'
            >
            > result.out:
            > $ cat result.out | hexdump -C
            > 00000000 c3 a2 c2 80 c2 98 0a |.......|
            > 00000007
            >
            > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
            > Everything works fine if it is not expr mapping or if I replace «"‘"» with
            > «"\u2018"». Same issue with opening double english quote (u201C) and second
            > level opening russian quote (u201E).

            See :help :map-<expr> and search CSI. This applies to 0x80 byte, too.

            map.txt:
            Note that there are some tricks to make special keys work and escape CSI bytes
            in the text. The |:map| command also does this, thus you must avoid that it
            is done twice. This does not work: >
            :imap <expr> <F3> "<Char-0x611B>"
            Because the <Char- sequence is escaped for being a |:imap| argument and then
            again for using <expr>. This does work: >
            :imap <expr> <F3> "\u611B"
            Using 0x80 as a single byte before other text does not work, it will be seen
            as a special key.


            What happened is

            :noremap! <expr> - "<Char-0x2018>"

            map rhs: " <e2> <80> <98> "
            |
            | When creating mapping 0x80 and 0x9b (CSI) are escaped.
            | <80> => <80> <fe> X
            v
            mapblock.m_str: " <e2> <80> <fe> X <98> "
            |
            | When mapping is used it is evaluated. (eval_map_expr())
            | In this case, eval('"\xe2\x80\xfeX\x98"').
            v
            evaluated result: <e2> <80> <fe> X <98>
            |
            | To insert to the typeahead buffer, 0x80 and 0x9b are escaped again.
            | Since the result is an invalid sequence it is interpreted as three
            | characters. (vim_strsave_escape_csi())
            | <e2> => U+00e2 => <c3> <a2>
            | <80> <fe> X => not changed because it is interpreted as
            | special key like <Left> (<80> k l)
            | <98> => U+0098 => <c2> <98>
            v
            typeahead: <c3> <a2> <80> <fe> X <c2> <98>
            |
            | unescaped and inserted to the buffer
            | <80> <fe> X => <80> => U+0080 => <c2> <80>
            v
            buffer: <c3> <a2> <c2> <80> <c2> <98>


            Perhaps this could be fixed by unescaping the rhs before evaluating it.
            But I'm not sure there is no side effect.

            diff --git a/src/getchar.c b/src/getchar.c
            --- a/src/getchar.c
            +++ b/src/getchar.c
            @@ -4491,37 +4491,49 @@
            static char_u *
            eval_map_expr(str, c)
            char_u *str;
            int c; /* NUL or typed character for abbreviation */
            {
            char_u *res;
            char_u *p;
            + char_u *expr;
            char_u *save_cmd;
            pos_T save_cursor;

            + expr = vim_strsave(str);
            + if (expr == NULL)
            + return NULL;
            + vim_unescape_csi(expr);
            +
            save_cmd = save_cmdline_alloc();
            if (save_cmd == NULL)
            + {
            + vim_free(expr);
            return NULL;
            + }

            /* Forbid changing text or using ":normal" to avoid most of the bad side
            * effects. Also restore the cursor position. */
            ++textlock;
            #ifdef FEAT_EX_EXTRA
            ++ex_normal_lock;
            #endif
            set_vim_var_char(c); /* set v:char to the typed character */
            save_cursor = curwin->w_cursor;
            - p = eval_to_string(str, NULL, FALSE);
            + p = eval_to_string(expr, NULL, FALSE);
            --textlock;
            #ifdef FEAT_EX_EXTRA
            --ex_normal_lock;
            #endif
            curwin->w_cursor = save_cursor;

            restore_cmdline_alloc(save_cmd);
            +
            + vim_free(expr);
            +
            if (p == NULL)
            return NULL;
            res = vim_strsave_escape_csi(p);
            vim_free(p);

            return res;
            }




            --
            Yukihiro Nakadaira - yukihiro.nakadaira@...

            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          Your message has been successfully submitted and would be delivered to recipients shortly.