Loading ...
Sorry, an error occurred while loading the content.
 

Memory is used up when I replace "\t0\n" to "\n"

Expand Messages
  • Aleafs
    Dear all, A bug seems appeared when I tried to delete the last column of a text file (700K bytes, 20K lines ). Memory is used up and the computer need to be
    Message 1 of 5 , Sep 9, 2009
      Dear all,

      A bug seems appeared when I tried to delete the last column of a text
      file (700K bytes, 20K lines ).

      Memory is used up and the computer need to be reboot when I oprate as
      below:

      :1,$ s/\t0\n/\n/g

      In the other hand, command like this works normal:

      :1,$ s/\t0$//g

      The version of my vim is 7.2. It can be reappeared on both Linux and
      Windows.

      Best regards

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_dev" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • John Beckett
      ... In a substitute, n means two different things: - In the pattern, it refers to a newline. - In the replacement, it refers to a null byte (8 zero bits). You
      Message 2 of 5 , Sep 10, 2009
        Aleafs wrote:
        > :1,$ s/\t0\n/\n/g

        In a substitute, \n means two different things:
        - In the pattern, it refers to a newline.
        - In the replacement, it refers to a null byte (8 zero bits).

        You can see this at ':help :s' by following the first two links.

        In a replacement, '\r' inserts a newline. Yes, it's strange, and
        no, I didn't try to work out why this issue caused the problem
        you report.

        John


        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_dev" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • Aleafs
        Thanks a lot. Expecting your conclusion ... --~--~---------~--~----~------------~-------~--~----~ You received this message from the vim_dev maillist. For
        Message 3 of 5 , Sep 10, 2009
          Thanks a lot. Expecting your conclusion


          On 9月10日, 下午4时13分, "John Beckett" <johnb.beck...@...> wrote:
          > Aleafs wrote:
          > > :1,$ s/\t0\n/\n/g
          >
          > In a substitute, \n means two different things:
          > - In the pattern, it refers to a newline.
          > - In the replacement, it refers to a null byte (8 zero bits).
          >
          > You can see this at ':help :s' by following the first two links.
          >
          > In a replacement, '\r' inserts a newline. Yes, it's strange, and
          > no, I didn't try to work out why this issue caused the problem
          > you report.
          >
          > John
          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_dev" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • Matt Wozniski
          ... Note that :% is a shortcut for :1,$ ... My guess: Vim is really, really bad at handling extremely long lines, and if the goal was to delete the last
          Message 4 of 5 , Sep 10, 2009
            On Thu, Sep 10, 2009 at 4:13 AM, John Beckett wrote:
            >
            > Aleafs wrote:
            >> :1,$  s/\t0\n/\n/g

            Note that :% is a shortcut for :1,$

            > In a substitute, \n means two different things:
            > - In the pattern, it refers to a newline.
            > - In the replacement, it refers to a null byte (8 zero bits).
            >
            > You can see this at ':help :s' by following the first two links.
            >
            > In a replacement, '\r' inserts a newline. Yes, it's strange, and
            > no, I didn't try to work out why this issue caused the problem
            > you report.

            My guess: Vim is really, really bad at handling extremely long lines,
            and if the goal was to delete "the last column of a text file" (I'm
            assuming every line ended with "\t0") then this command told vim to
            turn the file into one 700kb long line.

            ~Matt

            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_dev" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • Yakov Lerner
            ... 1. I don t understand why vim grows here (800MB and up)without giving memory overallocation error based on maxmem and maxmemtot . maxmem and
            Message 5 of 5 , Sep 15, 2009
              On Fri, Sep 11, 2009 at 06:57, Matt Wozniski <mjw@...> wrote:

              On Thu, Sep 10, 2009 at 4:13 AM, John Beckett wrote:
              >
              > Aleafs wrote:
              >> :1,$  s/\t0\n/\n/g

              Note that :% is a shortcut for :1,$

              > In a substitute, \n means two different things:
              > - In the pattern, it refers to a newline.
              > - In the replacement, it refers to a null byte (8 zero bits).
              >
              > You can see this at ':help :s' by following the first two links.
              >
              > In a replacement, '\r' inserts a newline. Yes, it's strange, and
              > no, I didn't try to work out why this issue caused the problem
              > you report.

              My guess: Vim is really, really bad at handling extremely long lines,
              and if the goal was to delete "the last column of a text file" (I'm
              assuming every line ended with "\t0") then this command told vim to
              turn the file into one 700kb long line.

              1. I don't understand why vim grows here (800MB and up)
              without giving memory overallocation error based on 'maxmem' and 'maxmemtot'. 
              'maxmem' and 'maxmemtot' have values of of 150524 (that's KB, or 150MB). 
              It's undo memory causes memory overuse here. 

              2. This problem of memory use here (and speed)is is with 'noswapfile' and 'undolevels', and it's solved by:

                  :set noswapfile undolevel=-1,

              Then vim finishes the replacement (wait 1/2 minutes).
              And vim opens 700k-long file consisting of single line and many 0-bytes, in the first place, without problem.

              Testcase:

              1) 700k file consisting of single line and many 0-bytes
              perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
              tr '\n' '\000' </tmp/1 >/tmp/2
              vim /tmp/2                 # no problem

              2) vim repalces newlines to 0-bytes without growing in memory, 700k file 20k lines:
              perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
              vim /tmp/1
              :set noswapfile undolevel=-1
              :%s/\n/\n/               " finished in 30 sec, resul tis single 700k-long line

              3) without 'et noswapfile undolevel=-1', blows up vim to pieces:
              perl -e 'printf "%8d %8d %8d %8d\n", $_,$_,$_,$_ for(1..20000)' >/tmp/1
              vim /tmp/1
              :%s/\n/\n/ " grows beyond 1G in size and killed by OS, if you have patience
              Killed

              In scenario #3, I thought vim would report error based on 'maxmem' and 'maxmemtot', but
              it did not. My values of 'maxmem' and 'maxmemtot' are 150254 (=150MB), but why
              vim grew over 800MB in vm size without giving memory size error ?

              Yakov
               

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_dev" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---

            Your message has been successfully submitted and would be delivered to recipients shortly.