Loading ...
Sorry, an error occurred while loading the content.

Questions about calculating memory usege by the vim undo feature

Expand Messages
  • Carlos Santos
    Hello, I m sending this question to the development group because it seems that only developers would be able to answer it. Moreover, it involves a possible
    Message 1 of 6 , May 5, 2014
    • 0 Attachment
      Hello,

      I'm sending this question to the development group because it seems that only developers would be able to answer it. Moreover, it involves a possible change request.

      I work for Red Hat as a software maintenance engineer. Recently one of our customers reported a crazy situation in which really BIG server ran out of memory due to VIM's undo feature. Some user in their environment did a global search and replace ( :%s/\n/,/g ) against a file that had 59 Million lines. The operation consumed more than 100 GB of memory and they had to reboot the machine in order to restore normal functionality.

      The problem is quite easy to reproduce. Even a small scale reproduction with a 46 Mb file and 6 Million lines consumed all available memory (12 Gb) of our test machine, until oom-killer killed it.

      We explained to the customer that they should not use VIM for such task. Anyway, there are two questions that I wold like to ask.

      1. Is there a formula that a single search-and-replace consumes x memory? Is this documented anywhere?

      2. Would it be reasonable to open a request to implement some kind of maximum cap as a safe guard?

      Thanks in advance,

      Carlos Santos (casantos)
      Senior *Software* Maintenance Engineer
      (no, I'm not going to fix your roof)
      Red Hat, Inc


      --
      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_dev" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
      For more options, visit https://groups.google.com/d/optout.
    • James McCoy
      ... There was a discussion[0] about a related issue recently. The specific case of “:%s/ n//” was addressed in 7.4.232, but a more general solution hasn t
      Message 2 of 6 , May 5, 2014
      • 0 Attachment
        On Mon, May 05, 2014 at 05:23:03PM -0400, Carlos Santos wrote:
        > I work for Red Hat as a software maintenance engineer. Recently one of
        > our customers reported a crazy situation in which really BIG server
        > ran out of memory due to VIM's undo feature. Some user in their
        > environment did a global search and replace ( :%s/\n/,/g ) against a
        > file that had 59 Million lines. The operation consumed more than 100
        > GB of memory and they had to reboot the machine in order to restore
        > normal functionality.

        There was a discussion[0] about a related issue recently. The specific
        case of “:%s/\n//” was addressed in 7.4.232, but a more general solution
        hasn't been implemented yet.

        [0]: http://mid.gmane.org/6c91658e-0df9-434d-bf65-8669aa400a76@...

        > 1. Is there a formula that a single search-and-replace consumes x memory? Is this documented anywhere?

        This specific category of replacement ("replace \n with ...") has
        pathological behavior, since it "touches" the entire buffer. Yukihiro
        commented in the above thread that “undo memory is about m * (((n + 1) *
        n / 2) - 1), where n is number of lines and m is line length”.

        There's a suggested patch in that thread and an item in the todo list,
        but nothing else yet.

        Cheers,
        --
        James
        GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy <jamessan@...>

        --
        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_dev" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
        For more options, visit https://groups.google.com/d/optout.
      • Antonio Giovanni Colombo
        Hi everybody, generally speaking, killing the offending process should have been enough, no need to reboot the machine. ... BEFORE modifying the interested
        Message 3 of 6 , May 5, 2014
        • 0 Attachment
          Hi everybody,

          generally speaking, killing the offending process should have been enough, no need to reboot the machine.

          I happen to edit huge files, and I usually type:

          :setlocal ul=-1

          BEFORE modifying the interested file. Of course, if the file is bigger than the available memory, paging will occur all the same, but one avoids problems with "undo".

          It does not solve the issue, but it could help...

          All the best, Antonio

          --
          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_dev" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
          For more options, visit https://groups.google.com/d/optout.
        • Christian Brabandt
          Hi Carlos! ... Out of curiosity, how much main memory was available? ... Isn t that more or less a use case of ulimit? BTW: For those cases, you might want to
          Message 4 of 6 , May 6, 2014
          • 0 Attachment
            Hi Carlos!

            On Mo, 05 Mai 2014, Carlos Santos wrote:

            > Hello,
            >
            > I'm sending this question to the development group because it seems that only developers would be able to answer it. Moreover, it involves a possible change request.
            >
            > I work for Red Hat as a software maintenance engineer. Recently one of our customers reported a crazy situation in which really BIG server ran out of memory due to VIM's undo feature. Some user in their environment did a global search and replace ( :%s/\n/,/g ) against a file that had 59 Million lines. The operation consumed more than 100 GB of memory and they had to reboot the machine in order to restore normal functionality.

            Out of curiosity, how much main memory was available?

            > The problem is quite easy to reproduce. Even a small scale reproduction with a 46 Mb file and 6 Million lines consumed all available memory (12 Gb) of our test machine, until oom-killer killed it.
            >
            > We explained to the customer that they should not use VIM for such task. Anyway, there are two questions that I wold like to ask.
            >
            > 1. Is there a formula that a single search-and-replace consumes x memory? Is this documented anywhere?
            >
            > 2. Would it be reasonable to open a request to implement some kind of maximum cap as a safe guard?

            Isn't that more or less a use case of ulimit? BTW: For those cases, you
            might want to consider the large file plugin (it resets the 'ul' option
            which should have prevented that crash).

            Best,
            Christian
            --
            Ab 50 denkt man beim Schuhe zubinden darüber nach, was man sonst noch
            alles erledigen kann, wenn man schon einmal hier unten ist.
            -- Horst Schrodt

            --
            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_dev" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
            For more options, visit https://groups.google.com/d/optout.
          • Carlos Santos
            ... 256GB (I told it was a big server). ... Yes, ulimit could prevent the crash but we can t prevent users from committing mistakes. :-) ... Carlos Santos
            Message 5 of 6 , May 6, 2014
            • 0 Attachment
              ----- Original Message -----
              > Hi Carlos!
              >
              > On Mo, 05 Mai 2014, Carlos Santos wrote:
              >
              > > Hello,
              > >
              > > I'm sending this question to the development group because it seems that
              > > only developers would be able to answer it. Moreover, it involves a
              > > possible change request.
              > >
              > > I work for Red Hat as a software maintenance engineer. Recently one of our
              > > customers reported a crazy situation in which really BIG server ran out of
              > > memory due to VIM's undo feature. Some user in their environment did a
              > > global search and replace ( :%s/\n/,/g ) against a file that had 59
              > > Million lines. The operation consumed more than 100 GB of memory and they
              > > had to reboot the machine in order to restore normal functionality.
              >
              > Out of curiosity, how much main memory was available?

              256GB (I told it was a big server).

              >
              > > The problem is quite easy to reproduce. Even a small scale reproduction
              > > with a 46 Mb file and 6 Million lines consumed all available memory (12
              > > Gb) of our test machine, until oom-killer killed it.
              > >
              > > We explained to the customer that they should not use VIM for such task.
              > > Anyway, there are two questions that I wold like to ask.
              > >
              > > 1. Is there a formula that a single search-and-replace consumes x memory?
              > > Is this documented anywhere?
              > >
              > > 2. Would it be reasonable to open a request to implement some kind of
              > > maximum cap as a safe guard?
              >
              > Isn't that more or less a use case of ulimit? BTW: For those cases, you
              > might want to consider the large file plugin (it resets the 'ul' option
              > which should have prevented that crash).

              Yes, ulimit could prevent the crash but we can't prevent users from committing mistakes. :-)

              > Best,
              > Christian

              Carlos Santos (casantos)
              Senior *Software* Maintenance Engineer
              (no, I'm not going to fix your roof)
              Red Hat, Inc

              --
              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_dev" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
              For more options, visit https://groups.google.com/d/optout.
            • Bram Moolenaar
              ... Linux is notoriously bad when running out of RAM. Perhaps you can find some workarounds somewhere, but making sure that processes don t use up all RAM is
              Message 6 of 6 , May 7, 2014
              • 0 Attachment
                Carlos Santos wrote:

                > I'm sending this question to the development group because it seems
                > that only developers would be able to answer it. Moreover, it involves
                > a possible change request.
                >
                > I work for Red Hat as a software maintenance engineer. Recently one of
                > our customers reported a crazy situation in which really BIG server
                > ran out of memory due to VIM's undo feature. Some user in their
                > environment did a global search and replace ( :%s/\n/,/g ) against a
                > file that had 59 Million lines. The operation consumed more than 100
                > GB of memory and they had to reboot the machine in order to restore
                > normal functionality.
                >
                > The problem is quite easy to reproduce. Even a small scale
                > reproduction with a 46 Mb file and 6 Million lines consumed all
                > available memory (12 Gb) of our test machine, until oom-killer killed
                > it.
                >
                > We explained to the customer that they should not use VIM for such
                > task. Anyway, there are two questions that I wold like to ask.
                >
                > 1. Is there a formula that a single search-and-replace consumes x
                > memory? Is this documented anywhere?
                >
                > 2. Would it be reasonable to open a request to implement some kind of
                > maximum cap as a safe guard?

                Linux is notoriously bad when running out of RAM. Perhaps you can find
                some workarounds somewhere, but making sure that processes don't use up
                all RAM is the only way to avoid trouble.

                You might want to switch to a BSD version, such as FreeBSD,they work
                much better when it comes to this kind of situation. But perhaps Red
                Hat isn't planning going that way :-).

                Thus limiting memory use is the only solution. We can (and will)
                improve memory use in Vim, but there will always be a way in which the
                user manages to to have Vim consume lots of memory.

                --
                hundred-and-one symptoms of being an internet addict:
                120. You ask a friend, "What's that big shiny thing?" He says, "It's the sun."

                /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                \\\ an exciting new programming language -- http://www.Zimbu.org ///
                \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                --
                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php

                ---
                You received this message because you are subscribed to the Google Groups "vim_dev" group.
                To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                For more options, visit https://groups.google.com/d/optout.
              Your message has been successfully submitted and would be delivered to recipients shortly.