Loading ...
Sorry, an error occurred while loading the content.

Re: Python items in the todo list

Expand Messages
  • Bram Moolenaar
    ... I think this implies we need to get and save the current directory for every Python command that is executed. Doesn t this add quite a bit of overhead? I
    Message 1 of 26 , May 17, 2013
    • 0 Attachment
      ZyX wrote:

      > > Interesting. So we would have a $VIMRUNTIME/python directory and we can
      > > put Python scripts there that we include with the distribution.
      > >
      > > I'm sure it is only a small step that users will ask to have a python
      > > directory in ~/.vim/python. Or more general, using 'runtimepath'.
      > >
      > > Then a plugin does not require to have its Python code in between a
      > > :python << EOF and EOF. That would be a lot nicer, right?
      > >
      > > What do others think?
      >
      > Automatically adding all items from `&rtp` to sys.path would be good.
      > It (and python3/) is already a standard directory in frawor for python
      > files.
      >
      > But I would vote against the patch to os.chdir implemented in python
      > and (as there is no better variant) for the solution based on
      > comparing current directories before and after `os.chdir`:
      >
      > 1. Implementation based on comparing current directories can be
      > written once and easily applied to all other interfaces.
      > 2. `os.chdir` is most common, but not the only way to change
      > directories from python: there are also at least `posix.chdir` and
      > calls to libc (e.g. indirectly from some bindings or directly using
      > ctypes, though I can’t imagine why the latter may be required).
      > 3. Proposed implementation will break once somebody deletes `os` from
      > sys.modules for some reason (I used to use this variant to reset `os`
      > module after mocking its methods for testing purposes; though as for
      > all non-builtin modules touching `sys.modules`).

      I think this implies we need to get and save the current directory for
      every Python command that is executed. Doesn't this add quite a bit of
      overhead? I originally asked for some kind of hook or callback, so that
      we can do something when Python changes directory at a lower level.

      I suppose we do need to wait with the Vim implementation of :cd until
      the Python command finishes. Thus os.chdir() would set a flag. Then
      when the Python command restores the directory we don't need to do
      anything. I would expect this to be quite common:
      saved_curdir = os.cwd()
      os.chdir(somewhere)
      ... do something with curdir ..
      os.chdir(saved_curdir)

      > About the implementation of `&rtp` handling: you can add there to
      > `sys.path` a special path like `'_vim_runtimepaths_'` and add hook to
      > `sys.path_hooks` that can handle it. Requires dropping python 2.2
      > support (`path_hooks` were added in python 2.3), but you won’t then
      > need to hack `options.c` to add or remove appropriate paths from
      > `sys.path` when changing `&rtp`.

      Do we need this kind of magic? Updating sys.path is much simpler and
      also does not hide what we are doing (in case someone is debugging
      what's going on). The only trick we need is when the user changes
      'runtimepath'. But that would happen very infrequently.

      --
      Arthur pulls Pin out. The MONK blesses the grenade as ...
      ARTHUR: (quietly) One, two, five ...
      GALAHAD: Three, sir!
      ARTHUR: Three.
      "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

      /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
      /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
      \\\ an exciting new programming language -- http://www.Zimbu.org ///
      \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Xavier de Gaye
      ... `os.chdir` and `posix.chdir` are two names that are bound to the same ... True `posix.chdir` is low level and is never used. The python documentation on
      Message 2 of 26 , May 18, 2013
      • 0 Attachment
        On Fri, May 17, 2013 at 9:07 PM, ZyX wrote:
        >> Interesting. So we would have a $VIMRUNTIME/python directory and we can
        >> put Python scripts there that we include with the distribution.
        >>
        >> I'm sure it is only a small step that users will ask to have a python
        >> directory in ~/.vim/python. Or more general, using 'runtimepath'.
        >>
        >> Then a plugin does not require to have its Python code in between a
        >> :python << EOF and EOF. That would be a lot nicer, right?
        >>
        >> What do others think?
        >
        > Automatically adding all items from `&rtp` to sys.path would be
        > good. It (and python3/) is already a standard directory in frawor
        > for python files.
        >
        > But I would vote against the patch to os.chdir implemented in python
        > and (as there is no better variant) for the solution based on
        > comparing current directories before and after `os.chdir`:
        >
        > 1. Implementation based on comparing current directories can be
        > written once and easily applied to all other interfaces.
        > 2. `os.chdir` is most common, but not the only way to change
        > directories from python: there are also at least `posix.chdir` and


        `os.chdir` and `posix.chdir` are two names that are bound to the same
        built-in function, as you can check with:

        >>> import os, posix
        >>> os.chdir is posix.chdir
        True

        `posix.chdir` is low level and is never used. The python documentation
        on 'posix' states:

        "Do not import this module directly. Instead, import the module os".

        `posix.chdir` is not even listed in the python documentation.


        > calls to libc (e.g. indirectly from some bindings or directly using
        > ctypes, though I can’t imagine why the latter may be required).


        This is kind of a red herring as the same issue already exists with
        the vimL libcall() function.


        > 3. Proposed implementation will break once somebody deletes `os`
        > from sys.modules for some reason (I used to use this variant to
        > reset `os` module after mocking its methods for testing purposes;
        > though as for all non-builtin modules touching `sys.modules`).


        It is not good practice to import a module twice (unless you own it
        and thus, know what you are doing) because importing a module may have
        side effects: when you import a module, you execute all the statements
        within the module, and executing them twice might not have been
        expected by the author of the module.

        If you mock some 'os' methods for testing purposes, then, when you are
        done with it, you must restore those methods instead of deleting the
        module from sys.modules and re-importing it a second time which is not
        safe.


        > About the implementation of `&rtp` handling: you can add there to
        > `sys.path` a special path like `'_vim_runtimepaths_'` and add hook
        > to `sys.path_hooks` that can handle it. Requires dropping python 2.2
        > support (`path_hooks` were added in python 2.3), but you won’t then
        > need to hack `options.c` to add or remove appropriate paths from
        > `sys.path` when changing `&rtp`.

        --
        Xavier

        Les Chemins de Lokoti: http://lokoti.alwaysdata.net

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • ZyX
        ... This does not matter. When you assign to `os.chdir` they are no longer identical. ... libcall() can use the same function for checking for os.chdir. ...
        Message 3 of 26 , May 18, 2013
        • 0 Attachment
          >>> Interesting. So we would have a $VIMRUNTIME/python directory and we can
          >>> put Python scripts there that we include with the distribution.
          >>>
          >>> I'm sure it is only a small step that users will ask to have a python
          >>> directory in ~/.vim/python. Or more general, using 'runtimepath'.
          >>>
          >>> Then a plugin does not require to have its Python code in between a
          >>> :python << EOF and EOF. That would be a lot nicer, right?
          >>>
          >>> What do others think?
          >>
          >> Automatically adding all items from `&rtp` to sys.path would be
          >> good. It (and python3/) is already a standard directory in frawor
          >> for python files.
          >>
          >> But I would vote against the patch to os.chdir implemented in python
          >> and (as there is no better variant) for the solution based on
          >> comparing current directories before and after `os.chdir`:
          >>
          >> 1. Implementation based on comparing current directories can be
          >> written once and easily applied to all other interfaces.
          >> 2. `os.chdir` is most common, but not the only way to change
          >> directories from python: there are also at least `posix.chdir` and
          >
          >
          >`os.chdir` and `posix.chdir` are two names that are bound to the same
          >built-in function, as you can check with:

          This does not matter. When you assign to `os.chdir` they are no longer
          identical.

          > >>> import os, posix
          > >>> os.chdir is posix.chdir
          > True
          >
          >`posix.chdir` is low level and is never used. The python documentation
          >on 'posix' states:
          >
          > "Do not import this module directly. Instead, import the module os".
          >
          >`posix.chdir` is not even listed in the python documentation.
          >
          >
          >> calls to libc (e.g. indirectly from some bindings or directly using
          >> ctypes, though I can’t imagine why the latter may be required).
          >
          >
          >This is kind of a red herring as the same issue already exists with
          >the vimL libcall() function.

          libcall() can use the same function for checking for os.chdir.

          >> 3. Proposed implementation will break once somebody deletes `os`
          >> from sys.modules for some reason (I used to use this variant to
          >> reset `os` module after mocking its methods for testing purposes;
          >> though as for all non-builtin modules touching `sys.modules`).
          >
          >
          >It is not good practice to import a module twice (unless you own it
          >and thus, know what you are doing) because importing a module may have
          >side effects: when you import a module, you execute all the statements
          >within the module, and executing them twice might not have been
          >expected by the author of the module.

          `os` seems to be safe regarding the issue.

          >If you mock some 'os' methods for testing purposes, then, when you are
          >done with it, you must restore those methods instead of deleting the
          >module from sys.modules and re-importing it a second time which is not
          >safe.

          Yes, I changed the implementation immediately after I found that removing from
          `sys.modules` may introduce strange TypeErrors (objects in module got assigned
          None). I have never seen such behavior for C modules though, only for python
          ones.

          >> About the implementation of `&rtp` handling: you can add there to
          >> `sys.path` a special path like `'_vim_runtimepaths_'` and add hook
          >> to `sys.path_hooks` that can handle it. Requires dropping python 2.2
          >> support (`path_hooks` were added in python 2.3), but you won’t then
          >> need to hack `options.c` to add or remove appropriate paths from
          >> `sys.path` when changing `&rtp`.

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • ZyX
          ... There are two other big advantages of this “magic”: user did not loose control over `sys.path` and it can be written in pure python. All advantages in
          Message 4 of 26 , May 18, 2013
          • 0 Attachment
            >Do we need this kind of magic? Updating sys.path is much simpler and
            >also does not hide what we are doing (in case someone is debugging
            >what's going on). The only trick we need is when the user changes
            >'runtimepath'. But that would happen very infrequently.

            There are two other big advantages of this “magic”: user did not loose control
            over `sys.path` and it can be written in pure python. All advantages in a list:
            it …

            1. makes it possible to specify which directories should be searched before
            plugin ones and which after;
            2. makes it possible to remove vim paths from sys.path completely and not ever
            have problems with vim adding them again;
            3. can be written in pure python and fallback implementation for old vims (the
            ones even without vim.eval) can be created;
            4. makes hacking non-if_py* C files not needed.

            There is a reason for coding it in C though (and porting @Xavier de Gaye file to C as well): this makes vim core functionality independent of presence of runtime files. There is a reason for not coding it in C as well: it is faster to write in python and even being written in python this will not have problems with performance (it’ll in any case waste much time on IO).

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          • Xavier de Gaye
            ... I think that it is reasonable to answer to anyone that complains something is broken after he deletes a module from sys.modules and re-import it again:
            Message 5 of 26 , May 19, 2013
            • 0 Attachment
              On Sat, May 18, 2013 at 4:12 PM, ZyX wrote:
              >>>> Interesting. So we would have a $VIMRUNTIME/python directory and we can
              >>>> put Python scripts there that we include with the distribution.
              >>>>
              >>>> I'm sure it is only a small step that users will ask to have a python
              >>>> directory in ~/.vim/python. Or more general, using 'runtimepath'.
              >>>>
              >>>> Then a plugin does not require to have its Python code in between a
              >>>> :python << EOF and EOF. That would be a lot nicer, right?
              >>>>
              >>>> What do others think?
              >>>
              >>> Automatically adding all items from `&rtp` to sys.path would be
              >>> good. It (and python3/) is already a standard directory in frawor
              >>> for python files.
              >>>
              >>> But I would vote against the patch to os.chdir implemented in python
              >>> and (as there is no better variant) for the solution based on
              >>> comparing current directories before and after `os.chdir`:
              >>>
              >>> 1. Implementation based on comparing current directories can be
              >>> written once and easily applied to all other interfaces.
              >>> 2. `os.chdir` is most common, but not the only way to change
              >>> directories from python: there are also at least `posix.chdir` and
              >>
              >>
              >>`os.chdir` and `posix.chdir` are two names that are bound to the same
              >>built-in function, as you can check with:
              >
              > This does not matter. When you assign to `os.chdir` they are no longer
              > identical.
              >
              >> >>> import os, posix
              >> >>> os.chdir is posix.chdir
              >> True
              >>
              >>`posix.chdir` is low level and is never used. The python documentation
              >>on 'posix' states:
              >>
              >> "Do not import this module directly. Instead, import the module os".
              >>
              >>`posix.chdir` is not even listed in the python documentation.
              >>
              >>
              >>> calls to libc (e.g. indirectly from some bindings or directly using
              >>> ctypes, though I can’t imagine why the latter may be required).
              >>
              >>
              >>This is kind of a red herring as the same issue already exists with
              >>the vimL libcall() function.
              >
              > libcall() can use the same function for checking for os.chdir.
              >
              >>> 3. Proposed implementation will break once somebody deletes `os`
              >>> from sys.modules for some reason (I used to use this variant to
              >>> reset `os` module after mocking its methods for testing purposes;
              >>> though as for all non-builtin modules touching `sys.modules`).
              >>
              >>
              >>It is not good practice to import a module twice (unless you own it
              >>and thus, know what you are doing) because importing a module may have
              >>side effects: when you import a module, you execute all the statements
              >>within the module, and executing them twice might not have been
              >>expected by the author of the module.
              >
              > `os` seems to be safe regarding the issue.
              >
              >>If you mock some 'os' methods for testing purposes, then, when you are
              >>done with it, you must restore those methods instead of deleting the
              >>module from sys.modules and re-importing it a second time which is not
              >>safe.
              >
              > Yes, I changed the implementation immediately after I found that removing from
              > `sys.modules` may introduce strange TypeErrors (objects in module got assigned
              > None). I have never seen such behavior for C modules though, only for python
              > ones.


              I think that it is reasonable to answer to anyone that complains
              something is broken after he deletes a module from sys.modules and
              re-import it again:

              "Please don't do that, if you really need to do it, then you are
              on your own".

              For example with the 'sys' module in the vim-python interface ('sys' is
              a C extension module by the way):

              :py3 import sys; del sys.modules['sys']
              :py3 import sys
              :py3 print(sys.stdout)
              Traceback (most recent call last):
              File "<string>", line 1, in <module>
              AttributeError: 'module' object has no attribute 'stdout'
              Press ENTER or type command to continue

              Re-binding a name such as done in the patch with os.chdir is actually
              quite common. It is even already done in the current vim-python
              interface with sys.stdout and sys.stderr (see PythonIO_Init_io()). A
              vim user writing a python program with the vim-python interface may
              also re-bind sys.stdout and sys.stderr to implement i/o redirection,
              but he is expected to restore the original bindings after he is done
              and certainly must not delete the 'sys' module and re-import it again.

              --
              Xavier

              Les Chemins de Lokoti: http://lokoti.alwaysdata.net

              --
              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_dev" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
              For more options, visit https://groups.google.com/groups/opt_out.
            Your message has been successfully submitted and would be delivered to recipients shortly.