Loading ...
Sorry, an error occurred while loading the content.

Re: capture() function to get output of command

Expand Messages
  • Ingo Karkat
    ... Yes, that s what I thought. Thanks for this data point! ... You bring up very good points; I fully agree. ... I think everyone agrees with this, and it
    Message 1 of 26 , May 7, 2013
    View Source
    • 0 Attachment
      On 07-May-2013 21:44 +0200, ZyX wrote:

      >> The functions in message.c write message with color attributes. So if splitting per line, it must get whole contents at the first.
      >> I guess someone want to get the contents as text which contains \n, but someone want to get the contents as array of lines. So I want to keep the contents as text.
      >>
      >> Thanks.
      >
      > Almost never used not splitted messages (they were only needed by me in case there is only one line of output; and only because splitting one line does not make much sense; code with proposed by @Ingo Karkat implementation of `capture()` is not a tiny bit harder to write in this case).

      Yes, that's what I thought. Thanks for this data point!

      > I have another proposal:

      You bring up very good points; I fully agree.

      > 1. Fix :redir to allow nesting.

      I think everyone agrees with this, and it should be first priority.

      > 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.

      I don't fully understand, but as long as it's backwards compatible,
      additional syntax, why not!?

      > 3. Make :redir automatically end when function ends if it holds reference to l:. In any case I doubt anybody needs to start redir in one function and end in another and do redirection to function-local variable at the time. This also must workaround stale :redir's left due to an exception (plugin authors almost always capture to function-local variables).

      This would alleviate a common programming error. Currently, all :redir
      commands have to be wrapped in try..finally to be safe, but few people
      do this.

      > 4. (optional) Add capture() function, but it should return a list of lists of dictionaries:
      >
      > [
      > [{'hl': "Error", 'text': "Error on first line"}, {'hl': "OtherHL", 'text': "Other highlight in first line (possible when using :echon)"}],
      > [{'hl': "Normal", 'text': "Regular message on second line"}],
      > ]
      >
      > . It is useful in some cases (like any use of :redir this cases happen mainly because of problems in API: distinguishing between `<LT>Space>` and `<Space>` in :map output (current code I use assumes nobody needs to define mapping like `<LT>F1>`; assumption has not proved false by the time, but having to use it bugs me); distinguishing between regular text and special characters in messages (don’t remember when I needed this, but there was some case)).

      Good idea. I had this need once with the :jumps output: Without the
      highlighting information, I had to implement a 99%-successful heuristic
      to decide whether the line refers to the current buffer or another file.
      It would be even better if the (mostly column-based) information issued
      by most commands came pre-parsed, but that probably would mean modifying
      all command implementations.

      > 5. (optional) Extend syntax of :redir to capture to a list of lines.

      Since this only applies for redirecting into a variable (not files or
      registers), I'd rather not further complicate the existing :redir
      command with this. Instead, if your proposal 4. capture() is
      implemented, have it take an argument {includeHl}, and just return a
      list of lines (without the Dictionary stuff) if 0 is passed. Most uses
      don't need the highlighting; this would make the function applicable to
      more use cases.


      I'd love to have all of those good enhancements, but I know that unless
      someone now starts working on a patch, this great discussion will again
      fizzle out, like so many before.

      -- regards, ingo

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • ZyX
      ... It is backwards compatible, but there is no additional syntax. When you call :redir with variable name it seems (I did not look at the code) to create a
      Message 2 of 26 , May 8, 2013
      View Source
      • 0 Attachment
        > > 1. Fix :redir to allow nesting.
        >
        > I think everyone agrees with this, and it should be first priority.
        >
        > > 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.
        >
        > I don't fully understand, but as long as it's backwards compatible,
        > additional syntax, why not!?

        It is backwards compatible, but there is no additional syntax. When you call :redir with variable name it seems (I did not look at the code) to create a variable, save variable name (but not the typval_T reference) and when writing to it use usual ways to get variable and write to the structure. Thus if :redir END is called when created variable is not in (script-)local scope of this call it errors out with “variable does not exist” or like. If :redir instead saved reference to the dict_T structure and then used usual methods of assigning dictionary key that error won’t appear. Alternatively this saved reference can be used to check whether :redir END is called in the same scope and do nothing (or give different error message) in case it is not. With nested :redir’ections I have nothing against error, without you will have to wrap `:redir =>` into a separate :try..:catch (with empty :catch block, and with pattern argument in this command) to be safe. In addition to surrounding :try..:finally of course.

        > > 3. Make :redir automatically end when function ends if it holds reference to l:. In any case I doubt anybody needs to start redir in one function and end in another and do redirection to function-local variable at the time. This also must workaround stale :redir's left due to an exception (plugin authors almost always capture to function-local variables).
        >
        > This would alleviate a common programming error. Currently, all :redir
        > commands have to be wrapped in try..finally to be safe, but few people
        > do this.
        >
        > > 4. (optional) Add capture() function, but it should return a list of lists of dictionaries:
        > >
        > > [
        > > [{'hl': "Error", 'text': "Error on first line"}, {'hl': "OtherHL", 'text': "Other highlight in first line (possible when using :echon)"}],
        > > [{'hl': "Normal", 'text': "Regular message on second line"}],
        > > ]
        > >
        > > . It is useful in some cases (like any use of :redir this cases happen mainly because of problems in API: distinguishing between `<LT>Space>` and `<Space>` in :map output (current code I use assumes nobody needs to define mapping like `<LT>F1>`; assumption has not proved false by the time, but having to use it bugs me); distinguishing between regular text and special characters in messages (don�t remember when I needed this, but there was some case)).
        >
        > Good idea. I had this need once with the :jumps output: Without the
        > highlighting information, I had to implement a 99%-successful heuristic
        > to decide whether the line refers to the current buffer or another file.
        > It would be even better if the (mostly column-based) information issued
        > by most commands came pre-parsed, but that probably would mean modifying
        > all command implementations.
        >
        > > 5. (optional) Extend syntax of :redir to capture to a list of lines.
        >
        > Since this only applies for redirecting into a variable (not files or
        > registers), I'd rather not further complicate the existing :redir
        > command with this. Instead, if your proposal 4. capture() is
        > implemented, have it take an argument {includeHl}, and just return a
        > list of lines (without the Dictionary stuff) if 0 is passed. Most uses
        > don't need the highlighting; this would make the function applicable to
        > more use cases.
        >
        > I'd love to have all of those good enhancements, but I know that unless
        > someone now starts working on a patch, this great discussion will again
        > fizzle out, like so many before.

        I am going to reduce the need for :redir for python users: further extension to python interface providing:

        - vim.maps/vim.buffer.maps (*iterable* and with output similar to maparg(lhs, mode, abbr, dict=True) (but as python object and .noremap with three values in place of two)): removes the need for parsing :redir|:map|:redir END;
        - vim.commands/vim.buffer.commands (iterable, after finished will be the only way to get command definition using something other then :redir);
        - vim.autocommands (iterable; same as above);
        - vim.signs (:sign [un]define/:sign list), vim.buffer.signs (:sign [un]place);
        - vim.registers (there is no way to get NUL out of the registers except for pasting register in a scratch buffer: any string manipulation transforms internal representation of NUL to "\n" and :redir|:reg truncates output in addition to having to guess (or much more complex: compare with `getreg()` output) whether ^@ meant NUL or somebody was typing ^ then @ like I did here).

        : this serves two purposes: make plugin writers prefer python (thus I am not talking about VimL API) and create normal API for the things that lack one. If you have other candidates tell me (I guess I already have one from you: :jumps). I will open a RFC here when API is fully designed; though I probably won’t have much time to write an implementation in next one or two months.

        Maybe I even won’t forget about the discussion after both API and implementation are finished. The thing I won’t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • Ingo Karkat
        ... y..:finally of course. Ah, thanks for the clarification; I think I understand now. I ve never done :redir END in a different scope, but it would be nice if
        Message 3 of 26 , May 8, 2013
        View Source
        • 0 Attachment
          On 08-May-2013 19:57 +0200, ZyX wrote:

          >>> 1. Fix :redir to allow nesting.
          >>
          >> I think everyone agrees with this, and it should be first priority.
          >>
          >>> 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.
          >>
          >> I don't fully understand, but as long as it's backwards compatible,
          >> additional syntax, why not!?
          >
          > It is backwards compatible, but there is no additional syntax. When you call :redir with variable name it seems (I did not look at the code) to create a variable, save variable name (but not the typval_T reference) and when writing to it use usual ways to get variable and write to the structure. Thus if :redir END is called when created variable is not in (script-)local scope of this call it errors out with “variable does not exist” or like. If :redir instead saved reference to the dict_T structure and then used usual methods of assigning dictionary key that error won’t appear. Alternatively this saved reference can be used to check whether :redir END is called in the same scope and do nothing (or give different error message) in case it is not. With nested :redir’ections I have nothing against error, without you will have to wrap `:redir =>` into a separate :try..:catch (with empty :catch block, and with pattern argument in this command) to be safe. In addition to surrounding :tr
          y..:finally of course.

          Ah, thanks for the clarification; I think I understand now. I've never
          done :redir END in a different scope, but it would be nice if this
          corner case bug could be fixed.

          >>> 3. Make :redir automatically end when function ends if it holds reference to l:. In any case I doubt anybody needs to start redir in one function and end in another and do redirection to function-local variable at the time. This also must workaround stale :redir's left due to an exception (plugin authors almost always capture to function-local variables).
          >>
          >> This would alleviate a common programming error. Currently, all :redir
          >> commands have to be wrapped in try..finally to be safe, but few people
          >> do this.
          >>
          >>> 4. (optional) Add capture() function, but it should return a list of lists of dictionaries:
          >>>
          >>> [
          >>> [{'hl': "Error", 'text': "Error on first line"}, {'hl': "OtherHL", 'text': "Other highlight in first line (possible when using :echon)"}],
          >>> [{'hl': "Normal", 'text': "Regular message on second line"}],
          >>> ]
          >>>
          >>> . It is useful in some cases (like any use of :redir this cases happen mainly because of problems in API: distinguishing between `<LT>Space>` and `<Space>` in :map output (current code I use assumes nobody needs to define mapping like `<LT>F1>`; assumption has not proved false by the time, but having to use it bugs me); distinguishing between regular text and special characters in messages (don�t remember when I needed this, but there was some case)).
          >>
          >> Good idea. I had this need once with the :jumps output: Without the
          >> highlighting information, I had to implement a 99%-successful heuristic
          >> to decide whether the line refers to the current buffer or another file.
          >> It would be even better if the (mostly column-based) information issued
          >> by most commands came pre-parsed, but that probably would mean modifying
          >> all command implementations.
          >>
          >>> 5. (optional) Extend syntax of :redir to capture to a list of lines.
          >>
          >> Since this only applies for redirecting into a variable (not files or
          >> registers), I'd rather not further complicate the existing :redir
          >> command with this. Instead, if your proposal 4. capture() is
          >> implemented, have it take an argument {includeHl}, and just return a
          >> list of lines (without the Dictionary stuff) if 0 is passed. Most uses
          >> don't need the highlighting; this would make the function applicable to
          >> more use cases.
          >>
          >> I'd love to have all of those good enhancements, but I know that unless
          >> someone now starts working on a patch, this great discussion will again
          >> fizzle out, like so many before.
          >
          > I am going to reduce the need for :redir for python users: further extension to python interface providing:
          >
          > - vim.maps/vim.buffer.maps (*iterable* and with output similar to maparg(lhs, mode, abbr, dict=True) (but as python object and .noremap with three values in place of two)): removes the need for parsing :redir|:map|:redir END;
          > - vim.commands/vim.buffer.commands (iterable, after finished will be the only way to get command definition using something other then :redir);
          > - vim.autocommands (iterable; same as above);
          > - vim.signs (:sign [un]define/:sign list), vim.buffer.signs (:sign [un]place);
          > - vim.registers (there is no way to get NUL out of the registers except for pasting register in a scratch buffer: any string manipulation transforms internal representation of NUL to "\n" and :redir|:reg truncates output in addition to having to guess (or much more complex: compare with `getreg()` output) whether ^@ meant NUL or somebody was typing ^ then @ like I did here).
          >
          > : this serves two purposes: make plugin writers prefer python (thus I am not talking about VimL API) and create normal API for the things that lack one. If you have other candidates tell me (I guess I already have one from you: :jumps). I will open a RFC here when API is fully designed; though I probably won’t have much time to write an implementation in next one or two months.

          I've seen your many recent Python patches, and it's certainly good to
          have more things exposed in a native language way. However, I'm a bit
          concerned if (but I know too little about your patches to be sure) you
          have to write all the plumbing from Vim's internal structures to the
          Python interface in a Python-specific way. I would prefer if most of the
          plumbing were done in Vim core, so that Vimscript as well as other
          language integrations could easily benefit. I imagine that the object
          model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
          that if all that access functionality existed in Vim core, each
          integration could get at that with a very thin language-dependent bridge.

          To put it in another way: Christian Brabandt recently implemented a
          Vimscript getsign() function and extended exists() to be able to access
          the sign information. I would hope that your vim.signs implementation
          could leverage most of that from Vim core. It would be bad if your
          implementation is a completely separate access of the core Vim
          structures, because of:
          - duplication = code bloat
          - other language integrations will eventually want to have the same
          access; each has to tediously reinvent the wheel, and will probably do
          so inconsistently
          - since maintenance is separate for each language, we end up with Python
          abilities for A, B, and C, Vimscript access for A & B (and use of :redir
          for C), and e.g. Ruby access only for C.

          > Maybe I even won’t forget about the discussion after both API and implementation are finished. The thing I won’t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.

          Don't get me wrong; I appreciate your efforts to improve the Python
          interface. I'm just a bit afraid that in the future, people will use
          Python just because there's no equally convenient way to access the
          information from Vimscript. To me, that would be bad. The Vimscript API
          should remain the first-class, full interface (that doesn't preclude
          other languages from having an equally rich interface), so that other
          languages are chosen based on their merits (mostly familiarity and
          external APIs that not available inside Vim), not based on need.

          -- regards, ingo

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • ZyX
          ... You can review the patches. Most code is converting arguments and result to/from PyObject* and error-checking, there is not much that can be moved to core.
          Message 4 of 26 , May 8, 2013
          View Source
          • 0 Attachment
            > I've seen your many recent Python patches, and it's certainly good to
            > have more things exposed in a native language way. However, I'm a bit
            > concerned if (but I know too little about your patches to be sure) you
            > have to write all the plumbing from Vim's internal structures to the
            > Python interface in a Python-specific way. I would prefer if most of the
            > plumbing were done in Vim core, so that Vimscript as well as other
            > language integrations could easily benefit. I imagine that the object
            > model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
            > that if all that access functionality existed in Vim core, each
            > integration could get at that with a very thin language-dependent bridge.

            You can review the patches. Most code is converting arguments and result to/from PyObject* and error-checking, there is not much that can be moved to core. I can’t generalize things like WindowAttr for e.g. lua: it cannot be used because WindowAttr returns PyObject* and in lua it would “return” void and push to the stack. Not saying I do not know other APIs. Things that make sense to put in core I put in core, but there are not much: in that sequence I only added functions for finding windows, temporary switching to specific window+tabpage and stricter get_option_value.

            > To put it in another way: Christian Brabandt recently implemented a
            > Vimscript getsign() function and extended exists() to be able to access
            > the sign information. I would hope that your vim.signs implementation
            > could leverage most of that from Vim core. It would be bad if your
            > implementation is a completely separate access of the core Vim
            > structures, because of:
            > - duplication = code bloat

            As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).

            > - other language integrations will eventually want to have the same
            > access; each has to tediously reinvent the wheel, and will probably do
            > so inconsistently

            Other languages’ integrations should be consistent with other languages’ best practices, not with python or VimL. Python users would not be happy with vim.lock(dict) in place of setattr(dict, 'locked', True) (dict.locked = True), but VimL uses first variant (with a command though). dict['abc']['def'] may be expected to throw an exception for absent 'abc' or 'def', or return undef and create a dictionary for key 'abc' (and even do this depending on options; I am talking about perl here, though dunno whether you can create a vim.dictionary there).

            > - since maintenance is separate for each language, we end up with Python
            > abilities for A, B, and C, Vimscript access for A & B (and use of :redir
            > for C), and e.g. Ruby access only for C.

            Yup. It would be good if there was a ruby user+developer to port my changes to ruby though. As I said most code is service one, lua won’t benefit from code that transforms to PyObject* handling errors in the middle, so you can expect extending lua being just as hard (or harder/easier depending on its C API) as extending python, with a small help from a small amount of functions that are general enough to have them in core.

            > > Maybe I even won�t forget about the discussion after both API and implementation are finished. The thing I won�t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.
            >
            > Don't get me wrong; I appreciate your efforts to improve the Python
            > interface. I'm just a bit afraid that in the future, people will use
            > Python just because there's no equally convenient way to access the
            > information from Vimscript. To me, that would be bad. The Vimscript API
            > should remain the first-class, full interface (that doesn't preclude
            > other languages from having an equally rich interface), so that other
            > languages are chosen based on their merits (mostly familiarity and
            > external APIs that not available inside Vim), not based on need.

            To choose other languages for their familiarity/*external* APIs somebody needs to find another person to write all the things I am implementing for Python and this would be good. I don’t think I will live long enough to see when you could choose language for vim plugin without taking vim<->language API into account. Also don’t forget that A and B that are not in ruby can be accessed through VimL layer.

            VimL being a first-class is not a good thing because of a big number of issues not fixable due to either backwards compatibility or too big codebase changes that require to fix them; if I was sure vim users are required to have some language but VimL I would immediately switch to it. Though you have a point about another languages: things have to be added to VimL because this way they automatically “appear” in other supported languages.

            Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can’t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          • Ingo Karkat
            ... Good, I m relieved. ... Yes, due to the mismatch between the C code in core and the mostly object-oriented APIs of other languages, I had assumed that
            Message 5 of 26 , May 8, 2013
            View Source
            • 0 Attachment
              On 08-May-2013 22:10 +0200, ZyX wrote:

              >> I've seen your many recent Python patches, and it's certainly good to
              >> have more things exposed in a native language way. However, I'm a bit
              >> concerned if (but I know too little about your patches to be sure) you
              >> have to write all the plumbing from Vim's internal structures to the
              >> Python interface in a Python-specific way. I would prefer if most of the
              >> plumbing were done in Vim core, so that Vimscript as well as other
              >> language integrations could easily benefit. I imagine that the object
              >> model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
              >> that if all that access functionality existed in Vim core, each
              >> integration could get at that with a very thin language-dependent bridge.
              >
              > You can review the patches. Most code is converting arguments and result to/from PyObject* and error-checking, there is not much that can be moved to core. I can’t generalize things like WindowAttr for e.g. lua: it cannot be used because WindowAttr returns PyObject* and in lua it would “return” void and push to the stack. Not saying I do not know other APIs. Things that make sense to put in core I put in core, but there are not much: in that sequence I only added functions for finding windows, temporary switching to specific window+tabpage and stricter get_option_value.

              Good, I'm relieved.

              >> To put it in another way: Christian Brabandt recently implemented a
              >> Vimscript getsign() function and extended exists() to be able to access
              >> the sign information. I would hope that your vim.signs implementation
              >> could leverage most of that from Vim core. It would be bad if your
              >> implementation is a completely separate access of the core Vim
              >> structures, because of:
              >> - duplication = code bloat
              >
              > As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).

              Yes, due to the mismatch between the C code in core and the mostly
              object-oriented APIs of other languages, I had assumed that there is
              quite some boilerplate code. If any of that could be wrapped for reuse
              [by other language integrations], either through macros or helper
              functions, that would be desirable.

              >> - other language integrations will eventually want to have the same
              >> access; each has to tediously reinvent the wheel, and will probably do
              >> so inconsistently
              >
              > Other languages’ integrations should be consistent with other languages’ best practices, not with python or VimL. Python users would not be happy with vim.lock(dict) in place of setattr(dict, 'locked', True) (dict.locked = True), but VimL uses first variant (with a command though). dict['abc']['def'] may be expected to throw an exception for absent 'abc' or 'def', or return undef and create a dictionary for key 'abc' (and even do this depending on options; I am talking about perl here, though dunno whether you can create a vim.dictionary there).
              >
              >> - since maintenance is separate for each language, we end up with Python
              >> abilities for A, B, and C, Vimscript access for A & B (and use of :redir
              >> for C), and e.g. Ruby access only for C.
              >
              > Yup. It would be good if there was a ruby user+developer to port my changes to ruby though. As I said most code is service one, lua won’t benefit from code that transforms to PyObject* handling errors in the middle, so you can expect extending lua being just as hard (or harder/easier depending on its C API) as extending python, with a small help from a small amount of functions that are general enough to have them in core.
              >
              >>> Maybe I even won�t forget about the discussion after both API and implementation are finished. The thing I won�t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.
              >>
              >> Don't get me wrong; I appreciate your efforts to improve the Python
              >> interface. I'm just a bit afraid that in the future, people will use
              >> Python just because there's no equally convenient way to access the
              >> information from Vimscript. To me, that would be bad. The Vimscript API
              >> should remain the first-class, full interface (that doesn't preclude
              >> other languages from having an equally rich interface), so that other
              >> languages are chosen based on their merits (mostly familiarity and
              >> external APIs that not available inside Vim), not based on need.
              >
              > To choose other languages for their familiarity/*external* APIs somebody needs to find another person to write all the things I am implementing for Python and this would be good. I don’t think I will live long enough to see when you could choose language for vim plugin without taking vim<->language API into account. Also don’t forget that A and B that are not in ruby can be accessed through VimL layer.
              >
              > VimL being a first-class is not a good thing because of a big number of issues not fixable due to either backwards compatibility or too big codebase changes that require to fix them; if I was sure vim users are required to have some language but VimL I would immediately switch to it. Though you have a point about another languages: things have to be added to VimL because this way they automatically “appear” in other supported languages.

              I've heard the dislike of Vimscript, but much of that is probably due to
              the developers' relative unfamiliarity with it. For extensions that are
              closely related to text-editing, I found it quite nice and powerful, and
              there is no mismatch between the Vim API and another language to bridge.
              But I agree that other stuff (e.g. any interfacing with external systems
              or networking stuff) is extremely hard (though people have written
              hashing functions or JSON parsers in Vimscript).

              > Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can’t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.

              Certainly, Vimscript won't win any performance contest; other language
              interpreters are probably way more optimized.


              Thanks for all your explanations; you indeed have very deep knowledge in
              this area. I'm looking forward to your patches and improved Python
              integration! Good luck!

              -- regards, ingo

              --
              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_dev" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
              For more options, visit https://groups.google.com/groups/opt_out.
            • ZyX
              ... To do this I need to know other languages APIs. And have to create a “header” file if_interp.h really containing C code like if_py_both.h (never added
              Message 6 of 26 , May 8, 2013
              View Source
              • 0 Attachment
                > > As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).
                >
                > Yes, due to the mismatch between the C code in core and the mostly
                > object-oriented APIs of other languages, I had assumed that there is
                > quite some boilerplate code. If any of that could be wrapped for reuse
                > [by other language integrations], either through macros or helper
                > functions, that would be desirable.

                To do this I need to know other languages APIs. And have to create a “header” file if_interp.h really containing C code like if_py_both.h (never added any files to vim except for tests and runtime ones). And have to write something that uses such functions in e.g. if_ruby.c: there is no other way I can prove this generalization necessary. Thus if some ruby developer will want to extend ruby in a similar way it would be he who should write generalization: I just do not have enough knowledge and time to do this.

                > I've heard the dislike of Vimscript, but much of that is probably due to
                > the developers' relative unfamiliarity with it. For extensions that are
                > closely related to text-editing, I found it quite nice and powerful, and
                > there is no mismatch between the Vim API and another language to bridge.
                > But I agree that other stuff (e.g. any interfacing with external systems
                > or networking stuff) is extremely hard (though people have written
                > hashing functions or JSON parsers in Vimscript).

                This is not a problem with other stuff. I have to care about a big bunch of options like &ignorecase. I have to always remember about strange defaults (:map without nore while best practices tell nore should be default, :command without -bar while most commands will be more convenient to use with it and so on). I have to remember about escaping, and correct escaping (three common escaping variants for different cases). I have to forget about writing some neat code because it does not work reliably (e.g. :!python %). I have to generate a big number of lines just to take care of NUL case when in python it is just a regular symbol. Even in zsh it looks like a regular symbol though zsh uses just the same NUL-terminated strings. I have to know about some inconsistencies which will never be fixed due to backwards compatibility. Do you know that there is no option for filename completion in user-defined commands except for `custom…`? I have to hope user have sane configuration. Did you ever hear python `subprocess` module refusing to work because user has wrong configuration options? Also try to write down rules for dividing a string into a list of statements for python and for VimL and compare it. There is no such thing like a bunch of variants of handling newline depending on the bunch of variants of non-immediately-surrounding context† in python.

                By the way, I have written both (adler32 hash (math was easier then base64 encoder/decoder which I used to write too) and both YAML (superset of JSON) and JSON parsers). And even a DSL for describing (and validating and completing) function and command arguments.

                Developers do not like VimL not because they do not know it. They dislike it because they do. I cannot say there are no things that bug me in python, zsh or perl (high-level languages I know good enough to judge, best known first), but VimL is the only one that bugs me that much. I have to say though that when I was writing some simple things it seemed good, but “the farther into the forest, the thicker the trees”. I just did not know all the cases where my simple things can break and did not care about them.

                † (NL in file, NL in file followed by spaces and backslash, NL in execute after commands not taking | as an argument, NL in execute after other commands, NL in execute after (built-in!) command taking expression (:echo) and unclosed quote, NL in execute after :function): main questions to answer are whether we are in a file, in a Ex prompt (BTW, have not explored this), in an :execute or in a function inside :execute. And what is the command we are computing arguments for.

                > > Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can�t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.
                >
                > Certainly, Vimscript won't win any performance contest; other language
                > interpreters are probably way more optimized.

                They are. Problem is not with optimizations, problem is some set of features like concurrent processing and a number of places where globals are used.

                > Thanks for all your explanations; you indeed have very deep knowledge in
                > this area. I'm looking forward to your patches and improved Python
                > integration! Good luck!

                --
                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php

                ---
                You received this message because you are subscribed to the Google Groups "vim_dev" group.
                To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                For more options, visit https://groups.google.com/groups/opt_out.
              • mattn
                On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote: I don t hope that capture() contains highlight attributes. It should be plain text. And I guess
                Message 7 of 26 , May 9, 2013
                View Source
                • 0 Attachment
                  On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
                  <snip>

                  I don't hope that capture() contains highlight attributes. It should be plain text. And I guess fixing redir nesting has problems.

                  1. There are plugins that depend on original behavior. This break compatible.
                  2. If skipped 'redir END' with exceptions, we can't close the nesting.

                  For about 2, one of solution is adding bang like 'redir! END'. but this will close all of redir nesting. So it's become un-expected behavior.

                  Thanks.
                  - Yasuhiro Matsumoto

                  --
                  --
                  You received this message from the "vim_dev" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php

                  ---
                  You received this message because you are subscribed to the Google Groups "vim_dev" group.
                  To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                  For more options, visit https://groups.google.com/groups/opt_out.
                • ZyX
                  ... Show me them then. It breaks only with nested redirections and it breaks only in the fashion that you get more expected output in place of error and empty
                  Message 8 of 26 , May 10, 2013
                  View Source
                  • 0 Attachment
                    пятница, 10 мая 2013 г., 6:09:15 UTC+4 пользователь mattn написал:
                    > On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
                    > <snip>
                    >
                    > I don't hope that capture() contains highlight attributes. It should be plain text. And I guess fixing redir nesting has problems.
                    >
                    > 1. There are plugins that depend on original behavior. This break compatible.

                    Show me them then. It breaks only with nested redirections and it breaks only in the fashion that you get more expected output in place of error and empty string in variable. There is only one case when nested :redir breaks in other fashion: a sequence of :redir's in one function without :redir ENDs except for last one. I have never seen such a strange thing, but there is a way to fix this case: make :redir update variable immediately each time new :redir is called without waiting for :redir END (in any case you need something to store text redirected at this level, why not make it store in variable/register/file? I do not think anybody benefits from "The variable will remain empty until redirection ends."). Unneeded :redirs will likely be GCed when function ends (third suggestion).

                    > 2. If skipped 'redir END' with exceptions, we can't close the nesting.

                    This is why I have 3-rd suggestion. Most of the time :redir is used to redirect to variables.

                    >
                    > For about 2, one of solution is adding bang like 'redir! END'. but this will close all of redir nesting. So it's become un-expected behavior.

                    You don't need :redir! END. Just don't leave all redirection options to user besides for redirection to variable inside a function. Or use :try..:finally.

                    If somebody has left stale redirections currently you will see unexpected behavior (error) with regular :redir, refer to the link in my first message. In addition to the text logged somewhere (but only to one place) until next :redir is called.

                    If somebody has left stale redirections with new proposal you will just have text logged somewhere (but only to one place as well) until session ends. It is very rare to see stale redirections: I have never seen them so far and never written something that leaves such redirections; and logging to some place with usual amount of messages is not too harmful. Thus advantages are greater then disadvantages.

                    To diagnose the issue :redir (without arguments) should print the stack of places where redirected text will go in order; it will only be actually written to the one last position.

                    >
                    > Thanks.
                    > - Yasuhiro Matsumoto

                    --
                    --
                    You received this message from the "vim_dev" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php

                    ---
                    You received this message because you are subscribed to the Google Groups "vim_dev" group.
                    To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                    For more options, visit https://groups.google.com/groups/opt_out.
                  • Ingo Karkat
                    On 10-May-2013 04:09 +0200, mattn wrote: I beg your pardon, but I find your entire argument weak and unconstructive. ... Both ZyX s and my recent messages have
                    Message 9 of 26 , May 13, 2013
                    View Source
                    • 0 Attachment
                      On 10-May-2013 04:09 +0200, mattn wrote:

                      I beg your pardon, but I find your entire argument weak and unconstructive.

                      > On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
                      > <snip>
                      >
                      > I don't hope that capture() contains highlight attributes. It should
                      > be plain text.

                      Both ZyX's and my recent messages have already shown cases where this
                      additional information is essential. If you don't need it, ignore the
                      information / don't request it (my suggestion makes this optional via a
                      passed flag).

                      > And I guess fixing redir nesting has problems.

                      Your intuitions and guesses aren't really helpful in this otherwise very
                      factual discussion.

                      > 1. There are plugins that depend on original behavior. This break compatible.

                      I cannot think of a plugin that depends on having a nested :redir throw
                      an error. That's just bad coding practice, and it could easily be
                      corrected in the plugin. Do you have an actual example? As I've shown,
                      contrarily there are plugins that (when working together) are indeed
                      broken due to what I would call an implementation deficiency.

                      > [3 lines deleted]

                      Please take no offense, I value your opinion like every other, but
                      please support them with facts and concrete examples as much as
                      possible. Thank you.

                      -- regards, ingo

                      --
                      --
                      You received this message from the "vim_dev" maillist.
                      Do not top-post! Type your reply below the text you are replying to.
                      For more information, visit http://www.vim.org/maillist.php

                      ---
                      You received this message because you are subscribed to the Google Groups "vim_dev" group.
                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                      For more options, visit https://groups.google.com/groups/opt_out.
                    • Ingo Karkat
                      ... ever hear python `subprocess` module refusing to work because user has wrong configuration options? Also try to write down rules for dividing a string into
                      Message 10 of 26 , May 13, 2013
                      View Source
                      • 0 Attachment
                        On 09-May-2013 00:54 +0200, ZyX wrote:

                        > > > [10 sentences deleted]
                        >
                        >> I've heard the dislike of Vimscript, but much of that is probably due to
                        >> the developers' relative unfamiliarity with it. For extensions that are
                        >> closely related to text-editing, I found it quite nice and powerful, and
                        >> there is no mismatch between the Vim API and another language to bridge.
                        >> But I agree that other stuff (e.g. any interfacing with external systems
                        >> or networking stuff) is extremely hard (though people have written
                        >> hashing functions or JSON parsers in Vimscript).
                        >
                        > This is not a problem with other stuff. I have to care about a big bunch of options like &ignorecase. I have to always remember about strange defaults (:map without nore while best practices tell nore should be default, :command without -bar while most commands will be more convenient to use with it and so on). I have to remember about escaping, and correct escaping (three common escaping variants for different cases). I have to forget about writing some neat code because it does not work reliably (e.g. :!python %). I have to generate a big number of lines just to take care of NUL case when in python it is just a regular symbol. Even in zsh it looks like a regular symbol though zsh uses just the same NUL-terminated strings. I have to know about some inconsistencies which will never be fixed due to backwards compatibility. Do you know that there is no option for filename completion in user-defined commands except for `custom…`? I have to hope user have sane configuration. Did you
                        ever hear python `subprocess` module refusing to work because user has wrong configuration options? Also try to write down rules for dividing a string into a list of statements for python and for VimL and compare it. There is no such thing like a bunch of variants of handling newline depending on the bunch of variants of non-immediately-surrounding context† in python.

                        Good explanation of the pain points. My take on this is: Vimscript is
                        made primarily for interactive use, and therefore it is difficult to
                        write watertight plugin code that works under all circumstances.

                        It's great to be able to write :! python % when you know the Python in
                        your PATH is the right one, and your file doesn't contain spaces, but in
                        a plugin, you have to pull together configuration variables,
                        executable(), fnamescape(), etc., and it gets messy quickly. Likewise,
                        :map is often right for ad-hoc mappings, but all plugins should use
                        :noremap. Same for 'ignorecase'.

                        > [2 sentences deleted]
                        >
                        > Developers do not like VimL not because they do not know it. They dislike it because they do. I cannot say there are no things that bug me in python, zsh or perl (high-level languages I know good enough to judge, best known first), but VimL is the only one that bugs me that much. I have to say though that when I was writing some simple things it seemed good, but “the farther into the forest, the thicker the trees”. I just did not know all the cases where my simple things can break and did not care about them.

                        For me, it's again the trade off (or historical accident) that was made
                        in favor of easy migration from ad-hoc macros to mappings to
                        full-fledged personal plugins to published plugins. The great thing is
                        that it's Vimscript all down that path, the downside is that what was a
                        simple one-liner has to be rewritten to a complex dozen-line function
                        (with stuff that has to be tediously learned) in order to be fully
                        robust under most users' customizations.

                        On the other hand, maybe we're holding ourselves to too high ideals.
                        Most plugins I've tried and reviewed (even from venerable long-time
                        contributors like Tim Pope or Dr. Chip, though admittedly much less so)
                        have certain deficiencies or break when used with obscure settings or
                        environments, but most people are happy to use them, anyway. So I tend
                        to defend Vimscript on its "worse is better" philosophy when language
                        purists try to bash it too much. But I would be happy if a different
                        approach (like your planned much-extended Python interface) turns out to
                        be as productive without so many pitfalls.

                        > [11 sentences deleted]

                        -- regards, ingo

                        --
                        --
                        You received this message from the "vim_dev" maillist.
                        Do not top-post! Type your reply below the text you are replying to.
                        For more information, visit http://www.vim.org/maillist.php

                        ---
                        You received this message because you are subscribed to the Google Groups "vim_dev" group.
                        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                        For more options, visit https://groups.google.com/groups/opt_out.
                      • ZyX
                        ... Refer to my message. Nested :redir throws an error only when context is switched, it is possible to have redir = commands command redir = mappings map
                        Message 11 of 26 , May 13, 2013
                        View Source
                        • 0 Attachment
                          > I cannot think of a plugin that depends on having a nested :redir throw
                          > an error.

                          Refer to my message. Nested :redir throws an error only when context is switched, it is possible to have

                          redir => commands
                          command
                          redir => mappings
                          map
                          redir => imappings
                          imap
                          " and so on
                          redir END

                          > That's just bad coding practice, and it could easily be
                          > corrected in the plugin.

                          I completely agree there. Also refer to my message, I explained there how this situation may easily be worked around (just making it always write to target when any :redir is issued is enough for plugins to work; finishing redirections to local variables after function finishes will delete stale redirs in all cases that make sense to write from my point of view and even not finishing should be rare enough and not result in much harm: at maximum vim killed by OOM killer or a few MiBs long file). I have never actually seen plugins with the above code though.

                          > Do you have an actual example? As I've shown,
                          > contrarily there are plugins that (when working together) are indeed
                          > broken due to what I would call an implementation deficiency.

                          --
                          --
                          You received this message from the "vim_dev" maillist.
                          Do not top-post! Type your reply below the text you are replying to.
                          For more information, visit http://www.vim.org/maillist.php

                          ---
                          You received this message because you are subscribed to the Google Groups "vim_dev" group.
                          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                          For more options, visit https://groups.google.com/groups/opt_out.
                        • mattn
                          ... I m thinking capture() is not replacement of redir. So I m thinking this issue should be separated with redir s issue. 1. The redir have a problem
                          Message 12 of 26 , Jun 16, 2013
                          View Source
                          • 0 Attachment
                            On Tuesday, May 14, 2013 5:46:25 AM UTC+9, ZyX wrote:
                            > > I cannot think of a plugin that depends on having a nested :redir throw
                            > > an error.
                            >
                            > Refer to my message. Nested :redir throws an error only when context is switched, it is possible to have
                            >
                            > redir => commands
                            > command
                            > redir => mappings
                            > map
                            > redir => imappings
                            > imap
                            > " and so on
                            > redir END
                            >
                            > > That's just bad coding practice, and it could easily be
                            > > corrected in the plugin.
                            >
                            > I completely agree there. Also refer to my message, I explained there how this situation may easily be worked around (just making it always write to target when any :redir is issued is enough for plugins to work; finishing redirections to local variables after function finishes will delete stale redirs in all cases that make sense to write from my point of view and even not finishing should be rare enough and not result in much harm: at maximum vim killed by OOM killer or a few MiBs long file). I have never actually seen plugins with the above code though.
                            >
                            > > Do you have an actual example? As I've shown,
                            > > contrarily there are plugins that (when working together) are indeed
                            > > broken due to what I would call an implementation deficiency.


                            I'm thinking capture() is not replacement of redir. So I'm thinking this issue should be separated with redir's issue.

                            1. The redir have a problem currently, But capture() doesn't.
                            2. Reasonable to implement.
                            3. Easy to use. Possible to write per one line.
                            4. I know there are some users who want capture().

                            I'm thinking it's good idea to solve some problem.

                            Thanks.

                            --
                            --
                            You received this message from the "vim_dev" maillist.
                            Do not top-post! Type your reply below the text you are replying to.
                            For more information, visit http://www.vim.org/maillist.php

                            ---
                            You received this message because you are subscribed to the Google Groups "vim_dev" group.
                            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                            For more options, visit https://groups.google.com/groups/opt_out.
                          • Shougo
                            ... Yes. I think so, too. 1. I can use capture() easily than :redir. Example: let foo = capture( :mes ) redir = foo mes redir END Which is the easy to
                            Message 13 of 26 , Jun 16, 2013
                            View Source
                            • 0 Attachment
                              2013年6月17日月曜日 11時49分10秒 UTC+9 mattn:
                              > On Tuesday, May 14, 2013 5:46:25 AM UTC+9, ZyX wrote:
                              > > > I cannot think of a plugin that depends on having a nested :redir throw
                              > > > an error.
                              > >
                              > > Refer to my message. Nested :redir throws an error only when context is switched, it is possible to have
                              > >
                              > > redir => commands
                              > > command
                              > > redir => mappings
                              > > map
                              > > redir => imappings
                              > > imap
                              > > " and so on
                              > > redir END
                              > >
                              > > > That's just bad coding practice, and it could easily be
                              > > > corrected in the plugin.
                              > >
                              > > I completely agree there. Also refer to my message, I explained there how this situation may easily be worked around (just making it always write to target when any :redir is issued is enough for plugins to work; finishing redirections to local variables after function finishes will delete stale redirs in all cases that make sense to write from my point of view and even not finishing should be rare enough and not result in much harm: at maximum vim killed by OOM killer or a few MiBs long file). I have never actually seen plugins with the above code though.
                              > >
                              > > > Do you have an actual example? As I've shown,
                              > > > contrarily there are plugins that (when working together) are indeed
                              > > > broken due to what I would call an implementation deficiency.
                              >
                              >
                              > I'm thinking capture() is not replacement of redir. So I'm thinking this issue should be separated with redir's issue.
                              >
                              > 1. The redir have a problem currently, But capture() doesn't.
                              > 2. Reasonable to implement.
                              > 3. Easy to use. Possible to write per one line.
                              > 4. I know there are some users who want capture().
                              >
                              > I'm thinking it's good idea to solve some problem.
                              >
                              > Thanks.

                              Yes. I think so, too.

                              1. I can use capture() easily than :redir.

                              Example:
                              let foo = capture(":mes")

                              redir => foo
                              mes
                              redir END

                              Which is the easy to understand?

                              2. And, the capture() is used for function arguments.

                              Example:
                              echo remote_expr("GVIM1", "capture(':mes')")

                              I think function is better than command to use it in Vim script.

                              3. I think the patch works well(because I tested it).
                              But to fix redir is no patches, it is too far to use it in my plugins.

                              --
                              --
                              You received this message from the "vim_dev" maillist.
                              Do not top-post! Type your reply below the text you are replying to.
                              For more information, visit http://www.vim.org/maillist.php

                              ---
                              You received this message because you are subscribed to the Google Groups "vim_dev" group.
                              To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                              For more options, visit https://groups.google.com/groups/opt_out.
                            Your message has been successfully submitted and would be delivered to recipients shortly.