Loading ...
Sorry, an error occurred while loading the content.

Encountered problem with changed Windows 'shcf' and 'sxq' defaults.

Expand Messages
  • Ingo Karkat
    Hello Vim developers, Sorry for touching that can of worms again, but I ve come upon a problem with the changed shellxquote option on Windows (patch 7.3.443
    Message 1 of 3 , Feb 22, 2013
    • 0 Attachment
      Hello Vim developers,

      Sorry for touching that can of worms again, but I've come upon a problem with
      the changed 'shellxquote' option on Windows (patch 7.3.443 (MS-Windows: 'shcf'
      and 'sxq' defaults are not very good) and the few patches following it. Here's
      the discussion as a refresher:
      http://groups.google.com/group/vim_dev/browse_thread/thread/bd53b29c5e5f2a50/be11dca5e9d501bb)

      The problem:

      #v+
      vim -N -u NONE
      :echo system('echo.hi(ho)hi')
      E484: Can't open file D:\temp\VIo408F.tmp
      #v-

      It's the unquoted parentheses in the command. My actual command was
      :echo system('icacls ' . shellescape('C:\foo') . ' /deny
      %userdomain%\%username%:(WD,AD,DC)')

      The problem lies in the way the command is passed to the shell, as the following
      experiments show:

      #v+
      REM This is how Vim passes the command with shcf=/c and sxq=(
      C:\> C:\Windows\system32\cmd.exe /c (echo.hi^(ho^)hi ^>D:\temp\VIo8945.tmp 2^>^&1)
      hi was unexpected at this time.

      REM Enclosing the entire parenthesized command in double quotes avoids the
      error, but breaks the redirection.
      C:\> C:\Windows\system32\cmd.exe /c "(echo.hi^(ho^)hi ^>D:\temp\VIo8945.tmp 2^>^&1)"
      hi(ho)hi >D:\temp\VIo8945.tmp 2>&1

      REM Enclosing the command without the redirection in double quotes and
      parentheses seems to work.
      C:\> C:\Windows\system32\cmd.exe /c "(echo.hi^(ho^)hi)" ^>D:\temp\VIo8945.tmp 2^>^&1

      REM Best recommendation is probably to always quote all arguments, even when not
      necessary in a plain shell.
      C:\> C:\Windows\system32\cmd.exe /c (echo.^"hi^(ho^)hi^" ^>D:\temp\VIo8945.tmp
      2^>^&1)
      #v-

      Therefore, every argument should be quoted; this works:
      :echo system('icacls ' . shellescape('C:\foo') . ' /deny ' .
      shellescape('%userdomain%\%username%:(WD,AD,DC)'))


      I know that the arcane quoting rules of cmd.exe are to blame for this mess. It
      was a real hassle to get from the E484 to the root cause of this (in Vim 7.3.000
      (with the old quoting implementation), my command worked fine); something I'd
      rather avert from others, so hopefully my findings will be helpful in some way.
      It seems like an intractable problem that has no perfect solution.

      #v+
      Windows 7 SP1 Ultimate 64-bit English

      VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Feb 20 2013 15:34:49)
      MS-Windows 64-bit GUI version with OLE support
      Included patches: 1-823
      Compiled by ingo@...
      Huge version with GUI. Features included (+) or not (-):
      #v-

      -- regards, ingo
      --
      -- Ingo Karkat -- /^-- /^-- /^-- /^-- /^-- /^-- http://ingo-karkat.de/ --
      -- http://vim.sourceforge.net/account/profile.php?user_id=9713 --

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Ben Fritz
      ... In other words, if you encounter errors passing arguments to the shell, try using shellescape, or try escaping things manually with ^. Your original
      Message 2 of 3 , Feb 22, 2013
      • 0 Attachment
        On Friday, February 22, 2013 3:27:41 AM UTC-6, Ingo Karkat wrote:
        > Hello Vim developers,
        >
        > Sorry for touching that can of worms again, but I've come upon a problem with
        > the changed 'shellxquote' option on Windows (patch 7.3.443 (MS-Windows: 'shcf'
        > and 'sxq' defaults are not very good) and the few patches following it. Here's
        > the discussion as a refresher:
        > http://groups.google.com/group/vim_dev/browse_thread/thread/bd53b29c5e5f2a50/be11dca5e9d501bb)
        >
        > The problem:
        >
        > #v+
        > vim -N -u NONE
        > :echo system('echo.hi(ho)hi')
        > E484: Can't open file D:\temp\VIo408F.tmp
        > #v-
        >
        > It's the unquoted parentheses in the command. My actual command was
        > :echo system('icacls ' . shellescape('C:\foo') . ' /deny
        > %userdomain%\%username%:(WD,AD,DC)')
        >
        > [SNIP]
        >
        > REM Best recommendation is probably to always quote all arguments, even when not
        > necessary in a plain shell.
        > C:\> C:\Windows\system32\cmd.exe /c (echo.^"hi^(ho^)hi^" ^>D:\temp\VIo8945.tmp
        > 2^>^&1)
        > #v-
        >
        > Therefore, every argument should be quoted; this works:
        > :echo system('icacls ' . shellescape('C:\foo') . ' /deny ' .
        > shellescape('%userdomain%\%username%:(WD,AD,DC)'))
        >

        In other words, if you encounter errors passing arguments to the shell, try
        using shellescape, or try escaping things manually with ^. Your original example
        is:

        :echo system('echo.hi(ho)hi')

        Either of these variants work:

        :echo system('echo.hi^(ho^)hi')
        :echo system('echo.'.shellescape('hi(ho)hi'))

        Alternatively, :help 'shellxquote' says that if it is "( then )" is appended.
        This gave me a hint, which I tested; this also works:

        :set shellxquote="(
        :echo system('echo.hi(ho)hi')

        What are you asking for exactly? A note in the help? Or some further update to
        the quoting of shell arguments on Windows? I think a note in the help should
        suffice. Where do you think it should go? Maybe we can add a note to E484 that
        on Windows it is sometimes caused by unescaped special characters, and to try
        using shellescape or a new value of shellxquote to fix it.

        Can of worms, indeed! I didn't realize how bad cmd.exe is until that series of patches.

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • Ingo Karkat
        ... Yes, this works in the toy example, but not in the general case. ... I was hoping for someone to have the magic 100% solution :-) Seriously, I now doubt it
        Message 3 of 3 , Feb 22, 2013
        • 0 Attachment
          On 22-Feb-13 18:32:13 +0100, Ben Fritz wrote:

          > On Friday, February 22, 2013 3:27:41 AM UTC-6, Ingo Karkat wrote:
          >> Hello Vim developers,
          >>
          >> Sorry for touching that can of worms again, but I've come upon a problem with
          >> the changed 'shellxquote' option on Windows (patch 7.3.443 (MS-Windows: 'shcf'
          >> and 'sxq' defaults are not very good) and the few patches following it. Here's
          >> the discussion as a refresher:
          >> http://groups.google.com/group/vim_dev/browse_thread/thread/bd53b29c5e5f2a50/be11dca5e9d501bb)
          >>
          >> The problem:
          >>
          >> #v+
          >> vim -N -u NONE
          >> :echo system('echo.hi(ho)hi')
          >> E484: Can't open file D:\temp\VIo408F.tmp
          >> #v-
          >>
          >> It's the unquoted parentheses in the command. My actual command was
          >> :echo system('icacls ' . shellescape('C:\foo') . ' /deny
          >> %userdomain%\%username%:(WD,AD,DC)')
          >>
          >> [SNIP]
          >>
          >> REM Best recommendation is probably to always quote all arguments, even when not
          >> necessary in a plain shell.
          >> C:\> C:\Windows\system32\cmd.exe /c (echo.^"hi^(ho^)hi^" ^>D:\temp\VIo8945.tmp
          >> 2^>^&1)
          >> #v-
          >>
          >> Therefore, every argument should be quoted; this works:
          >> :echo system('icacls ' . shellescape('C:\foo') . ' /deny ' .
          >> shellescape('%userdomain%\%username%:(WD,AD,DC)'))
          >>
          >
          > In other words, if you encounter errors passing arguments to the shell, try
          > using shellescape, or try escaping things manually with ^. Your original example
          > is:
          >
          > :echo system('echo.hi(ho)hi')
          >
          > Either of these variants work:
          >
          > :echo system('echo.hi^(ho^)hi')
          > :echo system('echo.'.shellescape('hi(ho)hi'))
          >
          > Alternatively, :help 'shellxquote' says that if it is "( then )" is appended.
          > This gave me a hint, which I tested; this also works:
          >
          > :set shellxquote="(
          > :echo system('echo.hi(ho)hi')

          Yes, this works in the toy example, but not in the general case.

          > What are you asking for exactly? A note in the help? Or some further update to
          > the quoting of shell arguments on Windows?

          I was hoping for someone to have the magic 100% solution :-) Seriously, I now
          doubt it can be perfected, so we have to live with what we have. I had the
          command in my test suite for a very long time, and I was just very surprised how
          (and in which strange ways) this was suddenly broken. Usually, there are very
          few regressions in Vim (thanks, BTW!) I also see other plugin writers struggling
          with escaping on Windows.

          > I think a note in the help should suffice. Where do you think it
          > should go? Maybe we can add a note to E484 that on Windows it is
          > sometimes caused by unescaped special characters, and to try using
          > shellescape or a new value of shellxquote to fix it.

          Yeah, that's a good idea. How about this:

          diff -r 8b86b69546a9 runtime/doc/message.txt
          --- a/runtime/doc/message.txt Wed Feb 20 21:26:00 2013 +0100
          +++ b/runtime/doc/message.txt Fri Feb 22 20:43:06 2013 +0100
          @@ -730,7 +730,11 @@
          Can't open file {filename}
          Can't read file {filename}

          -Vim cannot read a temporary file.
          +Vim cannot read a temporary file. Especially on Windows, this can be caused
          +by missing escaping of special characters for cmd.exe; the approach was
          +changed with patch 7.3.443. Try using |shellescape()| for all shell arguments
          +given to |system()|, or explicitly add escaping with ^. Also see
          +'shellxquote' and 'shellxescape'.

          *E464* >
          Ambiguous use of user-defined command

          -- regards, ingo

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        Your message has been successfully submitted and would be delivered to recipients shortly.