Loading ...
Sorry, an error occurred while loading the content.
 

Re: [BUG] Quote characters in expression mapping are replaced with garbage

Expand Messages
  • Bram Moolenaar
    ... Yes, that is wrong, each byte is used as a character. -- hundred-and-one symptoms of being an internet addict: 84. Books in your bookcase bear the names
    Message 1 of 4 , Jan 4, 2011
      ZyX wrote:

      > Consider the following command:
      >
      > vim -u NONE -c 'noremap! <expr> - "‘"' \
      > -c 'normal i-' \
      > -c 'wq! result.out'
      >
      > result.out:
      > $ cat result.out | hexdump -C
      > 00000000 c3 a2 c2 80 c2 98 0a |.......|
      > 00000007
      >
      > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
      > Everything works fine if it is not expr mapping or if I replace «"‘"» with
      > «"\u2018"». Same issue with opening double english quote (u201C) and second
      > level opening russian quote (u201E).

      Yes, that is wrong, each byte is used as a character.

      --
      hundred-and-one symptoms of being an internet addict:
      84. Books in your bookcase bear the names Bongo, WinSock and Inside OLE

      /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
      /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
      \\\ an exciting new programming language -- http://www.Zimbu.org ///
      \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • Peter Odding
      ... I encountered this bug* while trying to map and to insert smart quotes for my notes.vim plug-in. Fortunately I found a simple workaround (for my use
      Message 2 of 4 , Jan 4, 2011
        > Consider the following command:
        >
        > vim -u NONE -c 'noremap!<expr> - "‘"' \
        > -c 'normal i-' \
        > -c 'wq! result.out'
        >
        > result.out:
        > $ cat result.out | hexdump -C
        > 00000000 c3 a2 c2 80 c2 98 0a |.......|
        > 00000007
        >
        > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
        > Everything works fine if it is not expr mapping or if I replace «"‘"» with
        > «"\u2018"». Same issue with opening double english quote (u201C) and second
        > level opening russian quote (u201E).

        I encountered this bug* while trying to map ' and " to insert smart
        quotes for my notes.vim plug-in. Fortunately I found a simple workaround
        (for my use case at least): Don't embed the special characters directly
        in the <expr> mapping but call a function that returns the special
        characters instead (in my case the function is defined in a different
        script but I assume this doesn't matter).

        - Peter Odding

        *
        https://github.com/xolox/vim-notes/blob/3e4ec8fe4b2f68cdedc788f7c503308f618d4865/autoload.vim#L396

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • Yukihiro Nakadaira
        ... See :help :map- and search CSI. This applies to 0x80 byte, too. map.txt: Note that there are some tricks to make special keys work and escape CSI
        Message 3 of 4 , Jan 5, 2011
          ZyX wrote:
          > Consider the following command:
          >
          > vim -u NONE -c 'noremap!<expr> - "‘"' \
          > -c 'normal i-' \
          > -c 'wq! result.out'
          >
          > result.out:
          > $ cat result.out | hexdump -C
          > 00000000 c3 a2 c2 80 c2 98 0a |.......|
          > 00000007
          >
          > Tested on vim-7.3 from gentoo repos and on vim-7.3.94 (revision f987220caa57).
          > Everything works fine if it is not expr mapping or if I replace «"‘"» with
          > «"\u2018"». Same issue with opening double english quote (u201C) and second
          > level opening russian quote (u201E).

          See :help :map-<expr> and search CSI. This applies to 0x80 byte, too.

          map.txt:
          Note that there are some tricks to make special keys work and escape CSI bytes
          in the text. The |:map| command also does this, thus you must avoid that it
          is done twice. This does not work: >
          :imap <expr> <F3> "<Char-0x611B>"
          Because the <Char- sequence is escaped for being a |:imap| argument and then
          again for using <expr>. This does work: >
          :imap <expr> <F3> "\u611B"
          Using 0x80 as a single byte before other text does not work, it will be seen
          as a special key.


          What happened is

          :noremap! <expr> - "<Char-0x2018>"

          map rhs: " <e2> <80> <98> "
          |
          | When creating mapping 0x80 and 0x9b (CSI) are escaped.
          | <80> => <80> <fe> X
          v
          mapblock.m_str: " <e2> <80> <fe> X <98> "
          |
          | When mapping is used it is evaluated. (eval_map_expr())
          | In this case, eval('"\xe2\x80\xfeX\x98"').
          v
          evaluated result: <e2> <80> <fe> X <98>
          |
          | To insert to the typeahead buffer, 0x80 and 0x9b are escaped again.
          | Since the result is an invalid sequence it is interpreted as three
          | characters. (vim_strsave_escape_csi())
          | <e2> => U+00e2 => <c3> <a2>
          | <80> <fe> X => not changed because it is interpreted as
          | special key like <Left> (<80> k l)
          | <98> => U+0098 => <c2> <98>
          v
          typeahead: <c3> <a2> <80> <fe> X <c2> <98>
          |
          | unescaped and inserted to the buffer
          | <80> <fe> X => <80> => U+0080 => <c2> <80>
          v
          buffer: <c3> <a2> <c2> <80> <c2> <98>


          Perhaps this could be fixed by unescaping the rhs before evaluating it.
          But I'm not sure there is no side effect.

          diff --git a/src/getchar.c b/src/getchar.c
          --- a/src/getchar.c
          +++ b/src/getchar.c
          @@ -4491,37 +4491,49 @@
          static char_u *
          eval_map_expr(str, c)
          char_u *str;
          int c; /* NUL or typed character for abbreviation */
          {
          char_u *res;
          char_u *p;
          + char_u *expr;
          char_u *save_cmd;
          pos_T save_cursor;

          + expr = vim_strsave(str);
          + if (expr == NULL)
          + return NULL;
          + vim_unescape_csi(expr);
          +
          save_cmd = save_cmdline_alloc();
          if (save_cmd == NULL)
          + {
          + vim_free(expr);
          return NULL;
          + }

          /* Forbid changing text or using ":normal" to avoid most of the bad side
          * effects. Also restore the cursor position. */
          ++textlock;
          #ifdef FEAT_EX_EXTRA
          ++ex_normal_lock;
          #endif
          set_vim_var_char(c); /* set v:char to the typed character */
          save_cursor = curwin->w_cursor;
          - p = eval_to_string(str, NULL, FALSE);
          + p = eval_to_string(expr, NULL, FALSE);
          --textlock;
          #ifdef FEAT_EX_EXTRA
          --ex_normal_lock;
          #endif
          curwin->w_cursor = save_cursor;

          restore_cmdline_alloc(save_cmd);
          +
          + vim_free(expr);
          +
          if (p == NULL)
          return NULL;
          res = vim_strsave_escape_csi(p);
          vim_free(p);

          return res;
          }




          --
          Yukihiro Nakadaira - yukihiro.nakadaira@...

          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        Your message has been successfully submitted and would be delivered to recipients shortly.