Loading ...
Sorry, an error occurred while loading the content.
 

Re: Improving vim startup time for very large files

Expand Messages
  • Mike Williams
    ... Elapsed time is ~30s. Putting a profiler on VIM while it was writing the file it reported around ~5s CPU time driving the write to disk - the rest of it
    Message 1 of 9 , Jul 17, 2013
      On 17/07/2013 17:13, Benjamin Fritz wrote:
      > On Wed, Jul 17, 2013 at 10:49 AM, Mike Williams
      > <mike.williams@...> wrote:
      >>
      >> Does anyone have hard numbers? I have just loaded an ~900MB PDF file in ~7s
      >> (Win7 x64, 8GB, Core2Duo 2.3GHz), my normal VIM config (although I do have
      >> maxmem always set to maximum).
      >
      > Now try writing it. I suppose if Vim is only being used as a viewer
      > this might be a non-issue, but I discovered the problem when trying to
      > create a file with a huge number of lines to test how Vim responded
      > to...something. I don't exactly remember what I was trying to test,
      > only that I gave up on having Vim create the file and instead did it
      > using command-line tools (a huge pain on Windows), and then eventually
      > gave up on testing in general because Vim was taking so long to
      > manipulate the file.

      Elapsed time is ~30s. Putting a profiler on VIM while it was writing
      the file it reported around ~5s CPU time driving the write to disk - the
      rest of it is waiting for file IO to complete. So both reading and
      writing of large files is (not too surprisingly) IO bound and dependent
      on OS behaviour and current system usage (available memory for file
      cache, paging other apps and data out as required, etc.)

      The solution for handling this is a concurrent reading/writing threads
      with some new and interesting problems dealing with editing before the
      concurrent activities have finished. Given how atypical this issue is I
      would think there is not a great push to improve things.

      >> First time to load the file took an age
      >> (>40s) due to loading it off disk
      >
      > That sounds about right. Or maybe longer.

      Cold file caches are a pain in the a**e! ;-)

      >> - once it is in the OS file cache
      >> restarting vim to read the file was quick (I'd expect some delay with such a
      >> large file). Is this the sort of pattern you are seeing?
      >>
      >
      > I don't recall.
      >


      Mike
      --
      EXPERIENCE - experience is a wonderful thing. It enables you to
      recognise a mistake when you make it again.

      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Ben Fritz
      ... That s not what I saw. I let Vim run after doing :w for several minutes, and then force-killed it. My file was millions of very small lines (maybe empty, I
      Message 2 of 9 , Jul 17, 2013
        On Wednesday, July 17, 2013 11:54:59 AM UTC-5, Mike Williams wrote:
        > On 17/07/2013 17:13, Benjamin Fritz wrote:
        >
        > > On Wed, Jul 17, 2013 at 10:49 AM, Mike Williams
        >
        > > <mike.williams@...> wrote:
        >
        > >>
        >
        > >> Does anyone have hard numbers? I have just loaded an ~900MB PDF file in ~7s
        >
        > >> (Win7 x64, 8GB, Core2Duo 2.3GHz), my normal VIM config (although I do have
        >
        > >> maxmem always set to maximum).
        >
        > >
        >
        > > Now try writing it. I suppose if Vim is only being used as a viewer
        >
        > > this might be a non-issue, but I discovered the problem when trying to
        >
        > > create a file with a huge number of lines to test how Vim responded
        >
        > > to...something. I don't exactly remember what I was trying to test,
        >
        > > only that I gave up on having Vim create the file and instead did it
        >
        > > using command-line tools (a huge pain on Windows), and then eventually
        >
        > > gave up on testing in general because Vim was taking so long to
        >
        > > manipulate the file.
        >
        >
        >
        > Elapsed time is ~30s.

        That's not what I saw. I let Vim run after doing :w for several minutes, and then force-killed it.

        My file was millions of very small lines (maybe empty, I don't remember). I don't plan on trying again for now.

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • Ernie Rael
        ... On some systems it also depends on how it is written. Is it a single huge write, or lots of small ones, or something in between. When it s writing, vim
        Message 3 of 9 , Jul 17, 2013
          On 7/17/2013 9:54 AM, Mike Williams wrote:
          > ...
          > Elapsed time is ~30s. Putting a profiler on VIM while it was writing
          > the file it reported around ~5s CPU time driving the write to disk -
          > the rest of it is waiting for file IO to complete. So both reading
          > and writing of large files is (not too surprisingly) IO bound and
          > dependent on OS behaviour and current system usage (available memory
          > for file cache, paging other apps and data out as required, etc.)

          On some systems it also depends on how it is written. Is it a single
          huge write, or lots of small ones, or something in between.

          When it's writing, vim copies all the characters doesn't it? I wonder
          how well it would perform if the bytes were copied into a memory mapped
          file.

          -ernie

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • John Little
          ... Vim s syntax colouring is brilliant for viewing log files; simple enough for on-the-fly highlighting of the stuff one is interested in. ... A very long
          Message 4 of 9 , Jul 18, 2013
            On Wednesday, July 17, 2013 5:58:45 PM UTC+12, Ron Aaron wrote:
            > I (and my colleagues) often need to view extremely large log files (> 1G).From force of habit we use vim;

            Vim's syntax colouring is brilliant for viewing log files; simple enough for on-the-fly highlighting of the stuff one is interested in.

            > but vim takes a very long time to open huge files.

            A "very long time"? My vim on Kubuntu 13.04, on a 7 year old dual core Athlon, 4 GiB RAM, takes 30 s to load 1 GiB of C source, after clearing the OS disc cache. I mention this because there have been reports of slowness of vim running on "network shares", and maybe there's some such slowing you down.

            If you're just looking at the end of the file,

            tail -n 100000 large_file | vim -

            is quick and can be useful. (Shame vim won't open a loop device.)

            Regards, John Little

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          Your message has been successfully submitted and would be delivered to recipients shortly.