Loading ...
Sorry, an error occurred while loading the content.

Re: Writing large files is slow

Expand Messages
  • Bram Moolenaar
    ... Attached is the patch as I have it now. I fixed several warnings and made a few cosmetic changes. The core code changes looks good. I m not sure if the
    Message 1 of 17 , Mar 3, 2011
    • 0 Attachment
      Ivan Krasilnikov wrote:

      > On Tue, Mar 1, 2011 at 00:05, Bram Moolenaar <Bram@...> wrote:
      > > I had a quick look and it looks good.
      > >
      > > I think assert.h is not available everywhere. =C2=A0If you only use the
      > > assert() method, perhaps we can use a configure check and when assert.h
      > > is not available define it ourselves? =C2=A0This could be in a test.h inc=
      > lude
      > > file perhaps, so it can be reused in other unittests.
      >
      > vim.h already unconditionally includes assert.h. So if is not
      > available, vim wouldn't compile.
      >
      > Also, my BSD man page states that "assert macro appeared in Version 6
      > AT&T UNIX", which is the grand daddy of all the Unixes, so it's quite
      > safe to assume that assert.h is always present, even on very old
      > systems.

      Attached is the patch as I have it now. I fixed several warnings and
      made a few cosmetic changes. The core code changes looks good. I'm not
      sure if the tests cover enough of "normal usage" though.

      Let me know if anyone spots a problem with this before I send it out as
      a regular patch.

      -
      ARTHUR: A scratch? Your arm's off!
      BLACK KNIGHT: No, it isn't.
      ARTHUR: Well, what's that then?
      BLACK KNIGHT: I've had worse.
      The Quest for the Holy Grail (Monty Python)

      /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
      /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
      \\\ an exciting new programming language -- http://www.Zimbu.org ///
      \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • Ivan Krasilnikov
      ... For me, memfile_test linking fails with these errors: objects/if_xcmdsrv.o: In function `serverEventProc : src/if_xcmdsrv.c:1362: undefined reference to
      Message 2 of 17 , Mar 4, 2011
      • 0 Attachment
        On Fri, Mar 4, 2011 at 00:03, Bram Moolenaar <Bram@...> wrote:
        > Attached is the patch as I have it now.  I fixed several warnings and
        > made a few cosmetic changes.  The core code changes looks good.  I'm not
        > sure if the tests cover enough of "normal usage" though.
        >
        > Let me know if anyone spots a problem with this before I send it out as
        > a regular patch.

        For me, memfile_test linking fails with these errors:

        objects/if_xcmdsrv.o: In function `serverEventProc':
        src/if_xcmdsrv.c:1362: undefined reference to `serverConvert'
        src/if_xcmdsrv.c:1416: undefined reference to `serverConvert'
        src/if_xcmdsrv.c:1280: undefined reference to `serverConvert'
        src/if_xcmdsrv.c:1282: undefined reference to `server_to_input_buf'
        src/if_xcmdsrv.c:1284: undefined reference to `eval_client_expr_to_string'
        objects/if_xcmdsrv.o: In function `serverSendToVim':
        src/if_xcmdsrv.c:394: undefined reference to `eval_client_expr_to_string'
        src/if_xcmdsrv.c:407: undefined reference to `server_to_input_buf'

        In test77.in number of lines to be inserted is decreased to 200 000,
        which isn't enough. mf_hash_grow() is first called only after about
        1.5M lines are inserted.

        In src/Makefile there's a line with an extra space before a tab, and
        make depend section isn't updated.

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • Bram Moolenaar
        ... Perhaps this depends on a combination of features? ... With two million lines it just gets too slow. Is there a way to do this quicker? It appears it s
        Message 3 of 17 , Mar 6, 2011
        • 0 Attachment
          Ivan Krasilnikov wrote:

          > On Fri, Mar 4, 2011 at 00:03, Bram Moolenaar <Bram@...> wrote:
          > > Attached is the patch as I have it now.  I fixed several warnings and
          > > made a few cosmetic changes.  The core code changes looks good.  I'm not
          > > sure if the tests cover enough of "normal usage" though.
          > >
          > > Let me know if anyone spots a problem with this before I send it out as
          > > a regular patch.
          >
          > For me, memfile_test linking fails with these errors:
          >
          > objects/if_xcmdsrv.o: In function `serverEventProc':
          > src/if_xcmdsrv.c:1362: undefined reference to `serverConvert'
          > src/if_xcmdsrv.c:1416: undefined reference to `serverConvert'
          > src/if_xcmdsrv.c:1280: undefined reference to `serverConvert'
          > src/if_xcmdsrv.c:1282: undefined reference to `server_to_input_buf'
          > src/if_xcmdsrv.c:1284: undefined reference to `eval_client_expr_to_string'
          > objects/if_xcmdsrv.o: In function `serverSendToVim':
          > src/if_xcmdsrv.c:394: undefined reference to `eval_client_expr_to_string'
          > src/if_xcmdsrv.c:407: undefined reference to `server_to_input_buf'

          Perhaps this depends on a combination of features?

          > In test77.in number of lines to be inserted is decreased to 200 000,
          > which isn't enough. mf_hash_grow() is first called only after about
          > 1.5M lines are inserted.

          With two million lines it just gets too slow. Is there a way to do this
          quicker? It appears it's the loop to insert the lines that is slow,
          perhaps doing more lines at a time will help.

          > In src/Makefile there's a line with an extra space before a tab, and
          > make depend section isn't updated.

          I'll fix that.

          --
          If your life is a hard drive,
          Christ can be your backup.

          /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
          /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
          \\\ an exciting new programming language -- http://www.Zimbu.org ///
          \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • Lech Lorens
          ... I ve used the patch for some two days and it seems to work fine for me. That s with the vanilla Vim. However, the patch caused my heavily patched Vim to
          Message 4 of 17 , Mar 9, 2011
          • 0 Attachment
            On 3 March 2011 22:03, Bram Moolenaar <Bram@...> wrote:
            > Ivan Krasilnikov wrote:
            >
            > Attached is the patch as I have it now.  I fixed several warnings and
            > made a few cosmetic changes.  The core code changes looks good.  I'm not
            > sure if the tests cover enough of "normal usage" though.
            >
            > Let me know if anyone spots a problem with this before I send it out as
            > a regular patch.

            I've used the patch for some two days and it seems to work fine for me.
            That's with the vanilla Vim.

            However, the patch caused my heavily patched Vim to become very slow
            (It's not using
            CPU time, looks like it's waiting for a timeout) when executing
            external commands.
            Before I start debugging my patches would you have any suggestion
            where I should look first?

            --
            Cheers,
            Lech

            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          • Ivan Krasilnikov
            ... Errors disappears if I undefine FEAT_CLIENTSERVER. ... How about attached test? ... This patch is really only about hashing code in memfile.c. So, look at
            Message 5 of 17 , Mar 9, 2011
            • 0 Attachment
              On Sun, Mar 6, 2011 at 12:02, Bram Moolenaar <Bram@...> wrote:
              >> For me, memfile_test linking fails with these errors:
              >>
              >> objects/if_xcmdsrv.o: In function `serverEventProc':
              >> src/if_xcmdsrv.c:1362: undefined reference to `serverConvert'
              >> src/if_xcmdsrv.c:1416: undefined reference to `serverConvert'
              >> src/if_xcmdsrv.c:1280: undefined reference to `serverConvert'
              >> src/if_xcmdsrv.c:1282: undefined reference to `server_to_input_buf'
              >> src/if_xcmdsrv.c:1284: undefined reference to `eval_client_expr_to_string'
              >> objects/if_xcmdsrv.o: In function `serverSendToVim':
              >> src/if_xcmdsrv.c:394: undefined reference to `eval_client_expr_to_string'
              >> src/if_xcmdsrv.c:407: undefined reference to `server_to_input_buf'
              >
              > Perhaps this depends on a combination of features?

              Errors disappears if I undefine FEAT_CLIENTSERVER.

              >> In test77.in number of lines to be inserted is decreased to 200 000,
              >> which isn't enough. mf_hash_grow() is first called only after about
              >> 1.5M lines are inserted.
              >
              > With two million lines it just gets too slow.  Is there a way to do this
              > quicker?  It appears it's the loop to insert the lines that is slow,
              > perhaps doing more lines at a time will help.

              How about attached test?


              On Wed, Mar 9, 2011 at 12:20, Lech Lorens <lech.lorens@...> wrote:
              > Before I start debugging my patches would you have any suggestion
              > where I should look first?

              This patch is really only about hashing code in memfile.c. So, look at
              commits which touched that file.

              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php
            • Lech Lorens
              ... That seemed too easy and I already had tried this approach before asking for suggestions. The funny thing is at home on Linux I don t see any problems and
              Message 6 of 17 , Mar 12, 2011
              • 0 Attachment
                On 09-Mar-2011 Ivan Krasilnikov <infnty@...> wrote:
                > On Wed, Mar 9, 2011 at 12:20, Lech Lorens <lech.lorens@...> wrote:
                > > Before I start debugging my patches would you have any suggestion
                > > where I should look first?
                >
                > This patch is really only about hashing code in memfile.c. So, look at
                > commits which touched that file.

                That seemed too easy and I already had tried this approach before asking
                for suggestions. The funny thing is at home on Linux I don't see any
                problems and as a consequence can't do any debugging.

                Thanks for your help.

                --
                Cheers,
                Lech

                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php
              Your message has been successfully submitted and would be delivered to recipients shortly.