Loading ...
Sorry, an error occurred while loading the content.

NFA 3 times slower, syntime doesn't help

Expand Messages
  • David Fishburn
    NFA engine takes almost three times (300 vs 107 seconds) as long to complete my plugin task. ... VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 11 2013
    Message 1 of 5 , Oct 11, 2013
    • 0 Attachment

      NFA engine takes almost three times (300 vs 107 seconds) as long to complete my plugin task.

      :ver
      VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 11 2013 09:03:38)
      MS-Windows 32-bit GUI version with OLE support
      Included patches: 1-52

      My SQLUtilities plugin reformats SQL statements, through the use of many regular expressions using:
      :s/// 
      substitute()
      synID()
      syncIDtrans()
      search()

      I thought I would use :syntime to help identify what expressions were causing issues, but the results do not help me much.  I am also not certain if perhaps I an running things correctly to help identify where the issue might be.

      Basically, I do the following:
      :set re=x
      :syntime on
      my plugin call
      :syntime report
      paste to this email.

      :set re=x
      :syntime clear
      my plugin call
      :syntime report
      paste to this email.


      Here are the results:

      :set re=0

      With :syntime off, this takes 3 minute and 20 seconds.

        TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
        0.148406   2095   60      0.001223    0.000071  sqlNumber          -\=\<\d*\.\=[0-9_]\>
        0.014026   2035   0       0.000142    0.000007  sqlDashComment     --
        0.013963   2035   0       0.000078    0.000007  sqlMultiComment    /\*
        0.013940   2095   60      0.000152    0.000007  sqlString          '
        0.013811   2035   0       0.000200    0.000007  sqlSlashComment    \/\/
        0.002153   16919  16253   0.000002    0.000000  sqlString          "
        0.001525   7521   7521    0.000008    0.000000  sqlString          "
        0.000007   36     36      0.000000    0.000000  sqlString          '

        0.207830   34771


      :set re=1

      With :syntime off, this takes 1 minute and 47 seconds.

        TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
        0.295036   4295   120     0.001223    0.000069  sqlNumber          -\=\<\d*\.\=[0-9_]\>
        0.065744   4965   795     0.001454    0.000013  sqlString          '
        0.028317   4175   0       0.000233    0.000007  sqlDashComment     --
        0.028185   4175   0       0.000199    0.000007  sqlMultiComment    /\*
        0.027620   4175   0       0.000200    0.000007  sqlSlashComment    \/\/
        0.004863   34883  33451   0.000263    0.000000  sqlString          "
        0.003288   15763  15763   0.000012    0.000000  sqlString          "
        0.000011   72     72      0.000000    0.000000  sqlString          '

        0.453065   72503


      I don't even see many of the regexp expressions I used.  Given I am formatting SQL statements, I am searching for many SQL keywords, (SELECT, INSERT, UPDATE, DELETE, ...) and I don't see those words even referenced in the syntime report output.

      Thanks for any input.

      David

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
       
      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Julian Taylor
      the nfa engine seems to be badly suited for keyword type patterns, see also: https://groups.google.com/forum/#!topic/vim_dev/cPcMap1BdQw ... -- -- You received
      Message 2 of 5 , Oct 11, 2013
      • 0 Attachment
        the nfa engine seems to be badly suited for keyword type patterns, see also:
        https://groups.google.com/forum/#!topic/vim_dev/cPcMap1BdQw


        On Fri, Oct 11, 2013 at 4:26 PM, David Fishburn <dfishburn.vim@...> wrote:

        NFA engine takes almost three times (300 vs 107 seconds) as long to complete my plugin task.

        :ver
        VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 11 2013 09:03:38)
        MS-Windows 32-bit GUI version with OLE support
        Included patches: 1-52

        My SQLUtilities plugin reformats SQL statements, through the use of many regular expressions using:
        :s/// 
        substitute()
        synID()
        syncIDtrans()
        search()

        I thought I would use :syntime to help identify what expressions were causing issues, but the results do not help me much.  I am also not certain if perhaps I an running things correctly to help identify where the issue might be.

        Basically, I do the following:
        :set re=x
        :syntime on
        my plugin call
        :syntime report
        paste to this email.

        :set re=x
        :syntime clear
        my plugin call
        :syntime report
        paste to this email.


        Here are the results:

        :set re=0

        With :syntime off, this takes 3 minute and 20 seconds.

          TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
          0.148406   2095   60      0.001223    0.000071  sqlNumber          -\=\<\d*\.\=[0-9_]\>
          0.014026   2035   0       0.000142    0.000007  sqlDashComment     --
          0.013963   2035   0       0.000078    0.000007  sqlMultiComment    /\*
          0.013940   2095   60      0.000152    0.000007  sqlString          '
          0.013811   2035   0       0.000200    0.000007  sqlSlashComment    \/\/
          0.002153   16919  16253   0.000002    0.000000  sqlString          "
          0.001525   7521   7521    0.000008    0.000000  sqlString          "
          0.000007   36     36      0.000000    0.000000  sqlString          '

          0.207830   34771


        :set re=1

        With :syntime off, this takes 1 minute and 47 seconds.

          TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
          0.295036   4295   120     0.001223    0.000069  sqlNumber          -\=\<\d*\.\=[0-9_]\>
          0.065744   4965   795     0.001454    0.000013  sqlString          '
          0.028317   4175   0       0.000233    0.000007  sqlDashComment     --
          0.028185   4175   0       0.000199    0.000007  sqlMultiComment    /\*
          0.027620   4175   0       0.000200    0.000007  sqlSlashComment    \/\/
          0.004863   34883  33451   0.000263    0.000000  sqlString          "
          0.003288   15763  15763   0.000012    0.000000  sqlString          "
          0.000011   72     72      0.000000    0.000000  sqlString          '

          0.453065   72503


        I don't even see many of the regexp expressions I used.  Given I am formatting SQL statements, I am searching for many SQL keywords, (SELECT, INSERT, UPDATE, DELETE, ...) and I don't see those words even referenced in the syntime report output.

        Thanks for any input.

        David

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
         
        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
         
        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • Linda W
        ... Python never makes the top-REGEX engine lists -- can someone remind me why it was chosen? Was it purely a case of popularity winning over engineering
        Message 3 of 5 , Oct 12, 2013
        • 0 Attachment
          On 10/11/2013 7:26 AM, David Fishburn wrote:
          >
          > NFA engine takes almost three times (300 vs 107 seconds) as long to
          > complete my plugin task.
          ----
          Python never makes the top-REGEX engine lists -- can someone remind me
          why it was
          chosen? Was it purely a case of popularity winning over engineering
          design and benchmarks?

          From multiple articles the ones with most features and, fastest speed
          have been
          PCRE engines (perl compatible RegEx... THOUGH the default engine built
          into perl isn't among
          the fastest).

          Comparison websites put the PCRE-w/internal JIT (Just-In-Time state
          compiler).
          as the fastest by far: http://sljit.sourceforge.net/regex_perf.html

          Here is a comparison w/boost, showing boost in front -- but only against
          PCRE w/o JIT
          the PCRE w/JIT was 2-5x faster than the one without. So.. if Boost was
          only 3-5% faster
          than the no-JIT PCRE, its' in the dust w/the jit-version.

          As far as feature, dunno how up to date this is, but as of when it was
          written python didn't even have unicode properties (among other
          deficits)...
          http://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines

          Popularity, has never been a reliable bellwether for 'best'.



          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • Nikolay Pavlov
          ... complete my plugin task. ... why it was ... design and benchmarks? I do not get where you have seen a word about Python? AFAIK Bram did not choose any of
          Message 4 of 5 , Oct 13, 2013
          • 0 Attachment


            On Oct 13, 2013 10:05 AM, "Linda W" <vim@...> wrote:
            >
            > On 10/11/2013 7:26 AM, David Fishburn wrote:
            >>
            >>
            >> NFA engine takes almost three times (300 vs 107 seconds) as long to complete my plugin task.
            >
            > ----
            > Python never makes the top-REGEX engine lists -- can someone remind me why it was
            > chosen?  Was it purely a case of popularity winning over engineering design and benchmarks?

            I do not get where you have seen a word about Python? AFAIK Bram did not choose any of the existing engines and have written a new one.

            > From multiple articles the ones with most features and, fastest speed have been
            > PCRE engines (perl compatible RegEx... THOUGH the default engine built into perl isn't among
            > the fastest).
            > Comparison websites put the PCRE-w/internal JIT (Just-In-Time state compiler).
            > as the fastest by far:    http://sljit.sourceforge.net/regex_perf.html
            >
            > Here is a comparison w/boost, showing boost in front -- but only against PCRE w/o JIT
            > the PCRE w/JIT was 2-5x faster than the one without.  So.. if Boost was only 3-5% faster
            > than the no-JIT PCRE, its' in the dust w/the jit-version.
            >
            > As far as feature, dunno how up to date this is, but as of when it was
            > written python didn't even have unicode properties (among other
            > deficits)... http://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines
            >
            > Popularity, has never been a reliable bellwether for 'best'.
            >
            >
            >
            >
            > --
            > --
            > You received this message from the "vim_dev" maillist.
            > Do not top-post! Type your reply below the text you are replying to.
            > For more information, visit http://www.vim.org/maillist.php
            >
            > --- You received this message because you are subscribed to the Google Groups "vim_dev" group.
            > To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            > For more options, visit https://groups.google.com/groups/opt_out.

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
             
            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          • LCD 47
            On 12 October 2013, Linda W wrote: [...] ... [...] For what it s worth, PCRE doesn t have much in common with Perl aside from the syntax. It
            Message 5 of 5 , Oct 13, 2013
            • 0 Attachment
              On 12 October 2013, Linda W <vim@...> wrote:
              [...]
              > From multiple articles the ones with most features and, fastest speed
              > have been PCRE engines (perl compatible RegEx... THOUGH the default
              > engine built into perl isn't among the fastest).
              [...]

              For what it's worth, PCRE doesn't have much in common with Perl
              aside from the syntax. It was written from scratch by a Cambridge
              professor, out of frustration with the GNU regex. It's relatively
              recent code compared to Perl, the project was started about (IIRC) 7
              years ago, in relation to the Exim MTA.

              /lcd

              --
              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_dev" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
              For more options, visit https://groups.google.com/groups/opt_out.
            Your message has been successfully submitted and would be delivered to recipients shortly.