Loading ...
Sorry, an error occurred while loading the content.

Re: ex large file

Expand Messages
  • Ben Fritz
    ... So I suppose the OP should try (if they re on a 64 bit system): ex -s largefile -c %d _ -c wq If they re not on a 64-bit system, Vim can t open the file
    Message 1 of 13 , Jul 30, 2012
    • 0 Attachment
      On Monday, July 30, 2012 2:05:33 PM UTC-5, Dominique Pelle wrote:
      > Dominique Pellé wrote:
      >
      >
      >
      > > Enda wrote:
      >
      > >
      >
      > >> 'echo "" > filetowipe' would be a new file, I want to edit the same file.
      >
      > >>
      >
      > >> I don't want to crash my desktop again. I think the fact that ex creates
      >
      > >> .swp and .swo files might contribute to the problem.
      >
      > >>
      >
      > >> - Enda
      >
      > >
      >
      > >
      >
      > > Somehow %d (same 1,$d) is using a lot of memory
      >
      > > with big files.
      >
      > >
      >
      > > This is what I did:
      >
      > >
      >
      > > 1) Create a 150 Mb file (10 M lines) with:
      >
      > > $ yes 'this is a test' | sed 10000000q > test.txt
      >
      > >
      >
      > > 2) Start vim with: vim -u NONE test.txt
      >
      > > On my PC, this is using 202 Mb of virtual memory
      >
      > >
      >
      > > 3) :%d
      >
      > > Virtual memory usage jumps to 729 Mb
      >
      > >
      >
      > > My suspicion was that undo needs lots of memory.
      >
      > > Doing the same steps with :set ul=1 | %d
      >
      > > hence disabling undo saves a lot of memory but
      >
      > > memory usage still jumps from 202 Mb to 462Mb
      >
      > > when replace step 3) with:
      >
      > >
      >
      > > :set ul=-1 | %d
      >
      > >
      >
      > > I wonder what uses so much memory when undo
      >
      > > is disabled.
      >
      > >
      >
      > > I also profiled memory usage with...
      >
      > >
      >
      > > $ valgrind --tool=massif ./vim -u NONE
      >
      > >
      >
      > > ... and it gives the following usage at the peak of memory use
      >
      > > (using a 75 Mb test.txt file):
      >
      > >
      >
      > > --------------------------------------------------------------------------------
      >
      > > n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)
      >
      > >
      >
      > > 21 6,382,232,109 237,533,520 192,136,937 45,396,583 0
      >
      > > 80.89% (192,136,937B) (heap allocation functions) malloc/new/new[],
      >
      > > --alloc-fns, etc.
      >
      > > ->80.79% (191,902,967B) 0x810FE7B: lalloc (misc2.c:929)
      >
      > > | ->72.35% (171,861,290B) 0x810FD98: alloc (misc2.c:828)
      >
      > > | | ->40.45% (96,088,064B) 0x81F6471: mf_alloc_bhdr (memfile.c:951)
      >
      > > | | | ->40.45% (96,088,064B) 0x81F5B1B: mf_new (memfile.c:392)
      >
      > > | | | ->40.29% (95,703,040B) 0x80F1F14: ml_new_data (memline.c:3503)
      >
      > > | | | | ->40.29% (95,698,944B) 0x80F08C6: ml_append_int (memline.c:2751)
      >
      > > | | | | | ->40.29% (95,698,944B) 0x80F03E6: ml_append (memline.c:2525)
      >
      > > | | | | | ->40.29% (95,698,944B) 0x80C4A2D: readfile (fileio.c:2219)
      >
      > > | | | | | ->40.29% (95,698,944B) 0x804E0D8: open_buffer (buffer.c:140)
      >
      > > | | | | | ->40.29% (95,698,944B) 0x81EF6DB: create_windows (main.c:2632)
      >
      > > | | | | | ->40.29% (95,698,944B) 0x81ED22D: main (main.c:833)
      >
      > > | | | | |
      >
      > > | | | | ->00.00% (4,096B) in 1+ places, all below ms_print's
      >
      > > threshold (01.00%)
      >
      > > | | | |
      >
      > > | | | ->00.16% (385,024B) in 1+ places, all below ms_print's
      >
      > > threshold (01.00%)
      >
      > > | | |
      >
      > > | | ->31.58% (75,001,587B) 0x8110278: vim_strsave (misc2.c:1253)
      >
      > > | | | ->31.57% (74,998,440B) 0x8132825: op_yank (ops.c:2985)
      >
      > > | | | | ->31.57% (74,998,440B) 0x8130043: op_delete (ops.c:1728)
      >
      > > | | | | ->31.57% (74,998,440B) 0x80AF4F0: ex_operators (ex_docmd.c:8500)
      >
      > > | | | | ->31.57% (74,998,440B) 0x80A68CC: do_one_cmd (ex_docmd.c:2668)
      >
      > > | | | | ->31.57% (74,998,440B) 0x80A4194: do_cmdline (ex_docmd.c:1122)
      >
      > > | | | | ->31.57% (74,998,440B) 0x812710A: nv_colon (normal.c:5412)
      >
      > > | | | | ->31.57% (74,998,440B) 0x812070B: normal_cmd (normal.c:1193)
      >
      > > | | | | ->31.57% (74,998,440B) 0x81EDBA3: main_loop (main.c:1294)
      >
      > > | | | | ->31.57% (74,998,440B) 0x81ED5C0: main (main.c:998)
      >
      > >
      >
      > > I'll try to investigate more later.
      >
      > >
      >
      > > -- Dominique
      >
      >
      >
      >
      >
      > Ah of course, when doing %d Vim is storing the deleted text
      >
      > into the unnamed register so that's using lots of memory.
      >
      >
      >
      > I can delete the entire file without memory increase by
      >
      > disabling undo and using the black hole register "_ :
      >
      >
      >
      > :set ul=-1
      >
      > gg"_dG
      >
      >
      >

      So I suppose the OP should try (if they're on a 64 bit system):

      ex -s largefile -c '%d _' -c wq

      If they're not on a 64-bit system, Vim can't open the file so they should instead use (from Tony):

      ex -s -c 'saveas! largefile' -c q

      ...which will be faster and use less memory anyway, even on a 64-bit system.

      Or they could just use any of the several non-Vim solutions given.

      --
      You received this message from the "vim_use" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    Your message has been successfully submitted and would be delivered to recipients shortly.