Loading ...
Sorry, an error occurred while loading the content.

NUL characters in command output truncates system() results

Expand Messages
  • LCD 47
    ... prints a . The output of system(command) is truncated at the first NUL character in command s output. This affects all functions calling
    Message 1 of 4 , Aug 2, 2013
    • 0 Attachment
      The easy way to describe this is by an example:

      :echo system('printf "a\0b\n"')

      prints "a". The output of "system(command)" is truncated at the first
      NUL character in command's output.

      This affects all functions calling get_cmd_output(), including
      "make", "grep", and friends, and that's particularly annoying since some
      compilers (f.i. ghc-mod Haskell compiler) include NULs in their error
      messages.

      Now, looking at get_cmd_output() it's pretty clear why this happens.
      It isn't obvious how to fix that though, since once we're out of the
      said function we lose all information about the real length of command's
      output.

      /lcd

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Bram Moolenaar
      ... I suppose truncation is never useful. How about changing the NUL (0x00) into SOH (0x01)? SOH hardly ever appears, thus it can still be recognized as
      Message 2 of 4 , Aug 3, 2013
      • 0 Attachment
        lcd wrote:

        > The easy way to describe this is by an example:
        >
        > :echo system('printf "a\0b\n"')
        >
        > prints "a". The output of "system(command)" is truncated at the first
        > NUL character in command's output.
        >
        > This affects all functions calling get_cmd_output(), including
        > "make", "grep", and friends, and that's particularly annoying since some
        > compilers (f.i. ghc-mod Haskell compiler) include NULs in their error
        > messages.
        >
        > Now, looking at get_cmd_output() it's pretty clear why this happens.
        > It isn't obvious how to fix that though, since once we're out of the
        > said function we lose all information about the real length of command's
        > output.

        I suppose truncation is never useful. How about changing the NUL (0x00)
        into SOH (0x01)? SOH hardly ever appears, thus it can still be
        recognized as probably having been a NUL.

        --
        A fine is a tax for doing wrong. A tax is a fine for doing well.

        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
        /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
        \\\ an exciting new programming language -- http://www.Zimbu.org ///
        \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • LCD 47
        ... Seems like a good idea. Ok, patch below. /lcd diff -Nrc vim.d154f5cf7f53/runtime/doc/eval.txt vim/runtime/doc/eval.txt ... *************** ...
        Message 3 of 4 , Aug 3, 2013
        • 0 Attachment
          On 3 August 2013, Bram Moolenaar <Bram@...> wrote:
          >
          > lcd wrote:
          >
          > > The easy way to describe this is by an example:
          > >
          > > :echo system('printf "a\0b\n"')
          > >
          > > prints "a". The output of "system(command)" is truncated at the first
          > > NUL character in command's output.
          > >
          > > This affects all functions calling get_cmd_output(), including
          > > "make", "grep", and friends, and that's particularly annoying since some
          > > compilers (f.i. ghc-mod Haskell compiler) include NULs in their error
          > > messages.
          > >
          > > Now, looking at get_cmd_output() it's pretty clear why this happens.
          > > It isn't obvious how to fix that though, since once we're out of the
          > > said function we lose all information about the real length of command's
          > > output.
          >
          > I suppose truncation is never useful. How about changing the NUL (0x00)
          > into SOH (0x01)? SOH hardly ever appears, thus it can still be
          > recognized as probably having been a NUL.

          Seems like a good idea. Ok, patch below.

          /lcd


          diff -Nrc vim.d154f5cf7f53/runtime/doc/eval.txt vim/runtime/doc/eval.txt
          *** vim.d154f5cf7f53/runtime/doc/eval.txt Sat Aug 3 16:08:45 2013
          --- vim/runtime/doc/eval.txt Sat Aug 3 16:08:45 2013
          ***************
          *** 5884,5889 ****
          --- 5884,5890 ----
          < To make the result more system-independent, the shell output
          is filtered to replace <CR> with <NL> for Macintosh, and
          <CR><NL> with <NL> for DOS-like systems.
          + <Nul> characters are converted to \001 or CTRL-A.
          The command executed is constructed using several options:
          'shell' 'shellcmdflag' 'shellxquote' {expr} 'shellredir' {tmp} 'shellxquote'
          ({tmp} is an automatically generated file name).
          diff -Nrc vim.d154f5cf7f53/src/ascii.h vim/src/ascii.h
          *** vim.d154f5cf7f53/src/ascii.h Sat Aug 3 16:08:45 2013
          --- vim/src/ascii.h Sat Aug 3 16:08:45 2013
          ***************
          *** 22,27 ****
          --- 22,28 ----
          #define ROT13(c, a) (((((c) - (a)) + 13) % 26) + (a))

          #define NUL '\000'
          + #define SOH '\001'
          #define BELL '\007'
          #define BS '\010'
          #define TAB '\011'
          diff -Nrc vim.d154f5cf7f53/src/misc1.c vim/src/misc1.c
          *** vim.d154f5cf7f53/src/misc1.c Sat Aug 3 16:08:45 2013
          --- vim/src/misc1.c Sat Aug 3 16:08:45 2013
          ***************
          *** 10887,10893 ****
          --- 10887,10898 ----
          buffer = NULL;
          }
          else
          + {
          buffer[len] = NUL; /* make sure the buffer is terminated */
          + for (i = 0; i < len; i++)
          + if (buffer[i] == NUL)
          + buffer[i] = SOH;
          + }

          done:
          vim_free(tempname);

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • ZyX
          ... I would like to see readsystem() function in addition to system() one in order to cope with the problem in the plugins with the output similar to
          Message 4 of 4 , Aug 3, 2013
          • 0 Attachment
            > I suppose truncation is never useful. How about changing the NUL (0x00)
            > into SOH (0x01)? SOH hardly ever appears, thus it can still be
            > recognized as probably having been a NUL.

            I would like to see readsystem() function in addition to system() one in order to cope with the problem in the plugins with the output similar to readfile().

            The SOH/STX pair is known to be used as a replacement for bash \[\] or zsh %{%} in ipython (which uses readline) and thus may appear in source code. From my point of view using raw control characters in source code is a bad idea, but I have seen a number of developers that do not think so.

            It is also the reason why I want readsystem(): while I do not use something that writes binary data to stdout I still must handle the cases when developer put control characters (including NUL) in his source code which I get by reading `hg cat` or `hg diff` output. Currently I have to either use python in this case or use something like

            let tmpfile=tempname()
            call system('hg cat …'.(stridx(&shellredir, '%s')==-1?(&shellredir.tmpfile):(substitute(&shellredir, '%s', tmpfile, '')))
            let r=readfile(tmpfile)
            call delete(tmpfile)

            (:read and :! are not an options because they do not handle newlines correctly (correctly=without ever transforming anything like adding trailing newline when there is no or replacing \r\n with \n)).

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          Your message has been successfully submitted and would be delivered to recipients shortly.