Loading ...
Sorry, an error occurred while loading the content.

Issue 103 in vim: Python thread are not running in background

Expand Messages
  • vim@...
    Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 103 by degu...@gmail.com: Python thread are not running in background
    Message 1 of 18 , Jan 15, 2013
    • 0 Attachment
      Status: New
      Owner: ----
      Labels: Type-Defect Priority-Medium

      New issue 103 by degu...@...: Python thread are not running in
      background
      http://code.google.com/p/vim/issues/detail?id=103

      Hi,

      While trying to fix an issue on clang_complete plugin
      (https://github.com/Rip-Rip/clang_complete/issues/216), it appears that
      background threads created by the plugin are only running while vim execute
      python code. Which means that those threads do not execute while vim is
      idle.

      In 2004 Hari Krishna Dara had the same issue, which seemed to be resolved.
      The original thread with the test-case:
      http://marc.info/?l=vim&m=110029576819749&w=2

      If you try to run that test, you'll also notice that ":q" is not working,
      you'll need to kill vim!

      Vim version:
      vim 7.3.754 on ArchLinux x86_64 (using stock binary).

      Thanks!
      Xavier


      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • vim@...
      Comment #1 on issue 103 by zyx....@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Wondering whether
      Message 2 of 18 , Jan 15, 2013
      • 0 Attachment
        Comment #1 on issue 103 by zyx....@...: Python thread are not running
        in background
        http://code.google.com/p/vim/issues/detail?id=103

        Wondering whether it depends on the kind of code that is executed in the
        thread. I did successfully launch threads that are only sleeping and then
        incrementing variable in background on amd64 (and unsuccessfully on arm
        with the same code, observing the same behavior your report here).

        For this particular problem (and almost any of this kind) there is a
        workaround though: just use multiprocessing module (+ don’t forget about
        disabling all vim signal handlers in spawned process and terminating the
        process on VimLeave event in parent process).

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • vim@...
        Comment #2 on issue 103 by zyx....@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Similar to the one
        Message 3 of 18 , Jan 15, 2013
        • 0 Attachment
          Comment #2 on issue 103 by zyx....@...: Python thread are not running
          in background
          http://code.google.com/p/vim/issues/detail?id=103

          Similar to the one you linked in
          http://marc.info/?l=vim&m=110029576819749&w=2. And it in fact runs fine
          here (vim-7.3.762, amd64).

          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • vim@...
          Comment #3 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Spawning process
          Message 4 of 18 , Jan 15, 2013
          • 0 Attachment
            Comment #3 on issue 103 by degu...@...: Python thread are not running
            in background
            http://code.google.com/p/vim/issues/detail?id=103

            Spawning process instead of thread is not an option in my case…

            I just tried with a vim-7.3.762 and can still reproduce 100% of the time. I
            should have mentionned that this is with python2, and python3 is not
            compiled in.

            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          • vim@...
            Comment #4 on issue 103 by zyx....@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 I also use Python2
            Message 5 of 18 , Jan 15, 2013
            • 0 Attachment
              Comment #4 on issue 103 by zyx....@...: Python thread are not running
              in background
              http://code.google.com/p/vim/issues/detail?id=103

              I also use Python2 for the test, but I use both pythoninterp=dynamic and
              python3interp=dynamic on amd64 and AFAIR non-dynamic python2-only builds on
              arm. Maybe this matters.

              Why this is not an option? If caches that are talked about are per-process
              ones you can just move all libclang-related code to the separate process;
              you are not likely to have performance impact notable by user. And
              processes are forcefully terminatable, so “shutdown of vim takes several
              seconds” or “unable to quit vim when thread is running” issues should not
              appear.

              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php
            • vim@...
              Comment #5 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Yes you re right
              Message 6 of 18 , Jan 15, 2013
              • 0 Attachment
                Comment #5 on issue 103 by degu...@...: Python thread are not running
                in background
                http://code.google.com/p/vim/issues/detail?id=103

                Yes you're right about multithreading. I'm just afraid of the cost of
                sending completion results from the "completion process".

                But anyway, this is not related to this bug :).

                On which platform are you testing? I would guess Windows since
                pythoninterp=dynamic seems to only be available on this platform.

                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php
              • vim@...
                Comment #6 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 I just tried
                Message 7 of 18 , Jan 15, 2013
                • 0 Attachment
                  Comment #6 on issue 103 by degu...@...: Python thread are not running
                  in background
                  http://code.google.com/p/vim/issues/detail?id=103

                  I just tried manually disabling PY_CAN_RECURSE from if_python.c, and the
                  thread seems to be running in background. In clang_complete the correct
                  behaviour can also be seen.

                  --
                  You received this message from the "vim_dev" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php
                • vim@...
                  Comment #7 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 I did find the
                  Message 8 of 18 , Jan 15, 2013
                  • 0 Attachment
                    Comment #7 on issue 103 by degu...@...: Python thread are not running
                    in background
                    http://code.google.com/p/vim/issues/detail?id=103

                    I did find the culprit!

                    Reverting commit 7f10daa706bb solves the issue (the test-case is correclty
                    incrementing the variable in background). It also prevents vim to compile
                    when PY_CAN_RECURSE is not set.

                    It doesn't solve the problem of non-terminating threads, but that's also
                    another issue.

                    --
                    You received this message from the "vim_dev" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php
                  • vim@...
                    Comment #8 on issue 103 by zyx....@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 ... Nope. It is
                    Message 9 of 18 , Jan 16, 2013
                    • 0 Attachment
                      Comment #8 on issue 103 by zyx....@...: Python thread are not running
                      in background
                      http://code.google.com/p/vim/issues/detail?id=103

                      > On which platform are you testing? I would guess Windows since
                      > pythoninterp=dynamic seems to only be available on this platform.

                      Nope. It is available anywhere, but pythoninterp=dynamic is the only choice
                      if you want to have both python and python3 support in one binary. It is
                      Gentoo on amd64. And Ubuntu ARM (efika MX smartbook modification of Ubuntu,
                      stripped of almost any package that was there in a fresh installation).

                      > Reverting commit 7f10daa706bb solves the issue (the test-case is
                      > correclty incrementing the variable in background). It also prevents vim
                      > to compile when PY_CAN_RECURSE is not set.

                      Now when you mentioned it I remember I saw some (threading?) problems with
                      this commit on vim-dev. Did not came to my mind as I forgot about threads
                      as they are not working on ARM and are not forcefully terminatable. This is
                      also why I suggest processes to anyone wanting to use threads.

                      Though reverting commit is not an option: it itself is solving some issues.
                      Better to check why I have the code running correctly and you do not.

                      --
                      You received this message from the "vim_dev" maillist.
                      Do not top-post! Type your reply below the text you are replying to.
                      For more information, visit http://www.vim.org/maillist.php
                    • Jakson Alves de Aquino
                      ... Perhaps this problem is related with the one that I ve reported two moths ago: https://groups.google.com/forum/?fromgroups=#!topic/vim_dev/1B2habthlBs --
                      Message 10 of 18 , Jan 16, 2013
                      • 0 Attachment
                        On Wed, Jan 16, 2013 at 11:59 AM, <vim@...> wrote:
                        > Comment #8 on issue 103 by zyx....@...: Python thread are not running
                        > in background
                        > http://code.google.com/p/vim/issues/detail?id=103
                        >
                        >> On which platform are you testing? I would guess Windows since
                        >> pythoninterp=dynamic seems to only be available on this platform.
                        >
                        >
                        > Nope. It is available anywhere, but pythoninterp=dynamic is the only choice
                        > if you want to have both python and python3 support in one binary. It is
                        > Gentoo on amd64. And Ubuntu ARM (efika MX smartbook modification of Ubuntu,
                        > stripped of almost any package that was there in a fresh installation).
                        >
                        >> Reverting commit 7f10daa706bb solves the issue (the test-case is correclty
                        >> incrementing the variable in background). It also prevents vim to compile
                        >> when PY_CAN_RECURSE is not set.
                        >
                        >
                        > Now when you mentioned it I remember I saw some (threading?) problems with
                        > this commit on vim-dev. Did not came to my mind as I forgot about threads as
                        > they are not working on ARM and are not forcefully terminatable. This is
                        > also why I suggest processes to anyone wanting to use threads.
                        >
                        > Though reverting commit is not an option: it itself is solving some issues.
                        > Better to check why I have the code running correctly and you do not.

                        Perhaps this problem is related with the one that I've reported two moths ago:
                        https://groups.google.com/forum/?fromgroups=#!topic/vim_dev/1B2habthlBs

                        --
                        Jakson Alves de Aquino
                        Federal University of Ceará
                        Social Sciences Department
                        www.lepem.ufc.br/aquino.php

                        --
                        You received this message from the "vim_dev" maillist.
                        Do not top-post! Type your reply below the text you are replying to.
                        For more information, visit http://www.vim.org/maillist.php
                      • ZyX
                        Replies from mailing list are not added to bug tracker: http://code.google.com/p/vim/issues/detail?id=103. Use commenting there. -- You received this message
                        Message 11 of 18 , Jan 16, 2013
                        • 0 Attachment
                          Replies from mailing list are not added to bug tracker: http://code.google.com/p/vim/issues/detail?id=103. Use commenting there.

                          --
                          You received this message from the "vim_dev" maillist.
                          Do not top-post! Type your reply below the text you are replying to.
                          For more information, visit http://www.vim.org/maillist.php
                        • vim@...
                          Comment #9 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Maybe the bug
                          Message 12 of 18 , Jan 16, 2013
                          • 0 Attachment
                            Comment #9 on issue 103 by degu...@...: Python thread are not running
                            in background
                            http://code.google.com/p/vim/issues/detail?id=103

                            Maybe the bug depend on python's version? Mine is 2.7.3.

                            And yes I know that reverting the culprit commit is not really an option:
                            it does fix a bug. However, the fix may not be the correct one. Is there a
                            test-case for that bug to see if we can fix it in another way?

                            --
                            You received this message from the "vim_dev" maillist.
                            Do not top-post! Type your reply below the text you are replying to.
                            For more information, visit http://www.vim.org/maillist.php
                          • vim@...
                            Comment #10 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Hum, I m wondering
                            Message 13 of 18 , Jan 16, 2013
                            • 0 Attachment
                              Comment #10 on issue 103 by degu...@...: Python thread are not
                              running in background
                              http://code.google.com/p/vim/issues/detail?id=103

                              Hum, I'm wondering whether the GIL is not taken too much time.

                              If we read the description of PyEval_InitThreads(), it says "it is
                              guaranteed that the lock has been created and that the calling thread has
                              acquired it". Calling PyGILState_Ensure() also takes the GIL. Which means
                              that at init time, the GIL is taken 2 times, but only released once in
                              if_python.c:767 (by calling Python_SaveThread()). Since the GIL is never
                              released, that would explain why the background threads are not allowed to
                              run.

                              --
                              You received this message from the "vim_dev" maillist.
                              Do not top-post! Type your reply below the text you are replying to.
                              For more information, visit http://www.vim.org/maillist.php
                            • vim@...
                              Comment #11 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Changing the
                              Message 14 of 18 , Jan 16, 2013
                              • 0 Attachment
                                Comment #11 on issue 103 by degu...@...: Python thread are not
                                running in background
                                http://code.google.com/p/vim/issues/detail?id=103

                                Changing the initial value of pygilstate to PyGILState_LOCKED may be the
                                solution? (I'm only supposing since I don't know how the initial bug could
                                be triggered)

                                --
                                You received this message from the "vim_dev" maillist.
                                Do not top-post! Type your reply below the text you are replying to.
                                For more information, visit http://www.vim.org/maillist.php
                              • vim@...
                                Updates: Status: Accepted Comment #12 on issue 103 by lech.lor...@gmail.com: Python thread are not running in background
                                Message 15 of 18 , Jan 17, 2013
                                • 0 Attachment
                                  Updates:
                                  Status: Accepted

                                  Comment #12 on issue 103 by lech.lor...@...: Python thread are not
                                  running in background
                                  http://code.google.com/p/vim/issues/detail?id=103

                                  I also noticed the problem when using clang_complete. Last time closing Vim
                                  took 2 hours to complete! I'm using Vim 7.3.762 under Cygwin with Python
                                  2.6.8.

                                  --
                                  You received this message from the "vim_dev" maillist.
                                  Do not top-post! Type your reply below the text you are replying to.
                                  For more information, visit http://www.vim.org/maillist.php
                                • vim@...
                                  Comment #13 on issue 103 by jonathan...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 Here is how I
                                  Message 16 of 18 , Jan 25, 2013
                                  • 0 Attachment
                                    Comment #13 on issue 103 by jonathan...@...: Python thread are not
                                    running in background
                                    http://code.google.com/p/vim/issues/detail?id=103

                                    Here is how I patched my source.

                                    Attachments:
                                    vim-python-thread-issue.patch 974 bytes

                                    --
                                    --
                                    You received this message from the "vim_dev" maillist.
                                    Do not top-post! Type your reply below the text you are replying to.
                                    For more information, visit http://www.vim.org/maillist.php
                                  • vim@...
                                    Comment #14 on issue 103 by degu...@gmail.com: Python thread are not running in background http://code.google.com/p/vim/issues/detail?id=103 That patch seems
                                    Message 17 of 18 , Jan 29, 2013
                                    • 0 Attachment
                                      Comment #14 on issue 103 by degu...@...: Python thread are not
                                      running in background
                                      http://code.google.com/p/vim/issues/detail?id=103

                                      That patch seems to fix the issue indeed.

                                      --
                                      --
                                      You received this message from the "vim_dev" maillist.
                                      Do not top-post! Type your reply below the text you are replying to.
                                      For more information, visit http://www.vim.org/maillist.php

                                      ---
                                      You received this message because you are subscribed to the Google Groups "vim_dev" group.
                                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                                      For more options, visit https://groups.google.com/groups/opt_out.
                                    • vim@...
                                      Updates: Status: Fixed Comment #15 on issue 103 by brammool...@gmail.com: Python thread are not running in background
                                      Message 18 of 18 , Jan 30, 2013
                                      • 0 Attachment
                                        Updates:
                                        Status: Fixed

                                        Comment #15 on issue 103 by brammool...@...: Python thread are not
                                        running in background
                                        http://code.google.com/p/vim/issues/detail?id=103

                                        The patch is included as 7.3.786.
                                        Please reopen this issue if it didn't fix the problem.

                                        --
                                        --
                                        You received this message from the "vim_dev" maillist.
                                        Do not top-post! Type your reply below the text you are replying to.
                                        For more information, visit http://www.vim.org/maillist.php

                                        ---
                                        You received this message because you are subscribed to the Google Groups "vim_dev" group.
                                        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                                        For more options, visit https://groups.google.com/groups/opt_out.
                                      Your message has been successfully submitted and would be delivered to recipients shortly.