Loading ...
Sorry, an error occurred while loading the content.

Opening large files?

Expand Messages
  • Adrian Nagle
    m helping another user open a large, 3Gb, file. The standard windows editors balk, so I recommended VIM. Unfortunately, even vim crashes after scrolling
    Message 1 of 19 , Apr 16, 2014
    • 0 Attachment
      'm helping another user open a large, 3Gb, file. The standard windows
      editors balk, so I recommended VIM. Unfortunately, even vim crashes
      after scrolling some amount. For instance, he can't go straight to
      the end of file.

      The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
      any settings to modify to make vim more stable with large files, or is
      there some Windows performance limitation and just out of luck?

      Thanks.
      Adrian
      --
      Adrian Nagle (anagle@...)
      "There can be no thought of finishing, for 'aiming at the stars,' both
      literally and figuratively, is a problem to occupy generations, so
      that no matter how much progress one makes, there is always the thrill
      of just beginning." Dr. Goddard to H.G. Wells, 1932

      --
      --
      You received this message from the "vim_use" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_use" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
      For more options, visit https://groups.google.com/d/optout.
    • Tim Chase
      ... This is pretty common, so your system s specs should be able to handle it. Hints can be found here:
      Message 2 of 19 , Apr 16, 2014
      • 0 Attachment
        On 2014-04-16 09:58, Adrian Nagle wrote:
        > 'm helping another user open a large, 3Gb, file. The standard
        > windows editors balk, so I recommended VIM. Unfortunately, even
        > vim crashes after scrolling some amount. For instance, he can't go
        > straight to the end of file.
        >
        > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
        > any settings to modify to make vim more stable with large files, or
        > is there some Windows performance limitation and just out of luck?

        This is pretty common, so your system's specs should be able to
        handle it. Hints can be found here:

        http://vim.wikia.com/wiki/Faster_loading_of_large_files

        and a plugin/script to facilitate it can be found here:

        http://www.vim.org/scripts/script.php?script_id=1506

        -tim


        --
        --
        You received this message from the "vim_use" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_use" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
        For more options, visit https://groups.google.com/d/optout.
      • Dominique Pellé
        ... That plugin is definitely useful to speed up Vim with large files and to use less memory. Vim has to load the entire file in memory (unlike a few editors
        Message 3 of 19 , Apr 16, 2014
        • 0 Attachment
          Tim Chase <vim@...> wrote:

          > On 2014-04-16 09:58, Adrian Nagle wrote:
          >> 'm helping another user open a large, 3Gb, file. The standard
          >> windows editors balk, so I recommended VIM. Unfortunately, even
          >> vim crashes after scrolling some amount. For instance, he can't go
          >> straight to the end of file.
          >>
          >> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
          >> any settings to modify to make vim more stable with large files, or
          >> is there some Windows performance limitation and just out of luck?
          >
          > This is pretty common, so your system's specs should be able to
          > handle it. Hints can be found here:
          >
          > http://vim.wikia.com/wiki/Faster_loading_of_large_files
          >
          > and a plugin/script to facilitate it can be found here:
          >
          > http://www.vim.org/scripts/script.php?script_id=1506
          >
          > -tim


          That plugin is definitely useful to speed up Vim with large files
          and to use less memory.

          Vim has to load the entire file in memory (unlike a few editors
          that can load on demand). However, I would not expect crashes
          or running out of memory on a 3GB file with 32GB of RAM (and
          possibly more of virtual memory).

          Which version of Vim are you using?
          Can you get a stack trace where it crashes?
          Make sure you use the latest version of Vim to avoid wasting time
          on bugs possibly already fixed. I see a recent version of Vim
          for Windows here:

          http://sourceforge.net/projects/cream/files/Vim/

          Regards
          Dominique

          --
          --
          You received this message from the "vim_use" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_use" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
          For more options, visit https://groups.google.com/d/optout.
        • Ken Takata
          Hi, ... I wrote some patches to fix this, but they seem to be still unstable.
          Message 4 of 19 , Apr 17, 2014
          • 0 Attachment
            Hi,

            2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
            > 'm helping another user open a large, 3Gb, file. The standard windows
            > editors balk, so I recommended VIM. Unfortunately, even vim crashes
            > after scrolling some amount. For instance, he can't go straight to
            > the end of file.
            >
            > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
            > any settings to modify to make vim more stable with large files, or is
            > there some Windows performance limitation and just out of luck?

            There is a related item in the todo.txt:

            | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
            | pointer in long and seek offset in 64 bit var.

            I wrote some patches to fix this, but they seem to be still unstable.

            https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
            https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default

            Regards,
            Ken Takata

            --
            --
            You received this message from the "vim_use" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_use" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
            For more options, visit https://groups.google.com/d/optout.
          • Charles Campbell
            ... The LargeFile plugin turns the swapfile option for large files, so it would help avoid this problem; TIm has already provided a link to it, so I won t
            Message 5 of 19 , Apr 17, 2014
            • 0 Attachment
              Ken Takata wrote:
              > Hi,
              >
              > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
              >> 'm helping another user open a large, 3Gb, file. The standard windows
              >> editors balk, so I recommended VIM. Unfortunately, even vim crashes
              >> after scrolling some amount. For instance, he can't go straight to
              >> the end of file.
              >>
              >> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
              >> any settings to modify to make vim more stable with large files, or is
              >> there some Windows performance limitation and just out of luck?
              > There is a related item in the todo.txt:
              >
              > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
              > | pointer in long and seek offset in 64 bit var.
              >
              > I wrote some patches to fix this, but they seem to be still unstable.
              >
              > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
              > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
              >
              The LargeFile plugin turns the swapfile option for large files, so it
              would help avoid this problem; TIm has already provided a link to it, so
              I won't repeat that here.

              Is your "another user" using a FAT32 partition? That filesystem is
              limited to 4G files, and so possibly the swapfile is exceeding that limit.

              Regards,
              Chip Campbell

              --
              --
              You received this message from the "vim_use" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_use" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
              For more options, visit https://groups.google.com/d/optout.
            • Tim Chase
              ... In addition to the LargeFile tips, for certain types of edits, using a stream editor such as sed will usually let you operate on arbitrary-sized files,
              Message 6 of 19 , Apr 17, 2014
              • 0 Attachment
                On 2014-04-16 09:58, Adrian Nagle wrote:
                > 'm helping another user open a large, 3Gb, file. The standard
                > windows editors balk, so I recommended VIM. Unfortunately, even
                > vim crashes after scrolling some amount. For instance, he can't go
                > straight to the end of file.
                >
                > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                > any settings to modify to make vim more stable with large files, or
                > is there some Windows performance limitation and just out of luck?

                In addition to the LargeFile tips, for certain types of edits, using
                a stream editor such as "sed" will usually let you operate on
                arbitrary-sized files, as it only processes one line at a time
                (modulo what you keep in the hold-space). If you detail the types of
                transformations/changes that you are hoping to make, there might a
                way to do it that can be done regardless of the file-size.

                -tim


                --
                --
                You received this message from the "vim_use" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php

                ---
                You received this message because you are subscribed to the Google Groups "vim_use" group.
                To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                For more options, visit https://groups.google.com/d/optout.
              • Charles Campbell
                ... Sorry about the omission and typo: should be ...turns the swapfile option off... . And, instead of TIm , it should be Tim . Chip Campbell -- -- You
                Message 7 of 19 , Apr 17, 2014
                • 0 Attachment
                  Charles Campbell wrote:
                  > Ken Takata wrote:
                  >> Hi,
                  >>
                  >> 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                  >>> 'm helping another user open a large, 3Gb, file. The standard windows
                  >>> editors balk, so I recommended VIM. Unfortunately, even vim crashes
                  >>> after scrolling some amount. For instance, he can't go straight to
                  >>> the end of file.
                  >>>
                  >>> The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                  >>> any settings to modify to make vim more stable with large files, or is
                  >>> there some Windows performance limitation and just out of luck?
                  >> There is a related item in the todo.txt:
                  >>
                  >> | Win64: Seek error in swap file for a very big file (3 Gbyte). Check
                  >> storing
                  >> | pointer in long and seek offset in 64 bit var.
                  >>
                  >> I wrote some patches to fix this, but they seem to be still unstable.
                  >>
                  >> https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                  >>
                  >> https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                  >>
                  >>
                  > The LargeFile plugin turns the swapfile option for large files, so it
                  > would help avoid this problem; TIm has already provided a link to it,
                  > so I won't repeat that here.
                  >
                  > Is your "another user" using a FAT32 partition? That filesystem is
                  > limited to 4G files, and so possibly the swapfile is exceeding that
                  > limit.
                  >
                  Sorry about the omission and typo: should be "...turns the swapfile
                  option off...". And, instead of "TIm", it should be "Tim".

                  Chip Campbell

                  --
                  --
                  You received this message from the "vim_use" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php

                  ---
                  You received this message because you are subscribed to the Google Groups "vim_use" group.
                  To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                  For more options, visit https://groups.google.com/d/optout.
                • Adrian Nagle
                  Folks, I appreciate the help. I will look into the suggestions. We use NTFS. Thanks! Adrian -- -- You received this message from the vim_use maillist. Do
                  Message 8 of 19 , Apr 17, 2014
                  • 0 Attachment

                    Folks,

                    I appreciate the help.  I will look into the suggestions.  We use NTFS.

                    Thanks!

                    Adrian

                    --
                    --
                    You received this message from the "vim_use" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php

                    ---
                    You received this message because you are subscribed to the Google Groups "vim_use" group.
                    To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                    For more options, visit https://groups.google.com/d/optout.
                  • Tim Chase
                    ... No problem, CHip ;-) -tim -- -- You received this message from the vim_use maillist. Do not top-post! Type your reply below the text you are replying to.
                    Message 9 of 19 , Apr 17, 2014
                    • 0 Attachment
                      On 2014-04-17 09:44, Charles Campbell wrote:
                      > And, instead of "TIm", it should be "Tim".

                      No problem, CHip ;-)

                      -tim




                      --
                      --
                      You received this message from the "vim_use" maillist.
                      Do not top-post! Type your reply below the text you are replying to.
                      For more information, visit http://www.vim.org/maillist.php

                      ---
                      You received this message because you are subscribed to the Google Groups "vim_use" group.
                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                      For more options, visit https://groups.google.com/d/optout.
                    • Bram Moolenaar
                      ... Did you make progress on this? Can we also add some tests to verify the fix? -- ... /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net
                      Message 10 of 19 , Apr 29, 2014
                      • 0 Attachment
                        Ken Takata wrote:

                        > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                        > > 'm helping another user open a large, 3Gb, file. The standard windows
                        > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                        > > after scrolling some amount. For instance, he can't go straight to
                        > > the end of file.
                        > >
                        > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                        > > any settings to modify to make vim more stable with large files, or is
                        > > there some Windows performance limitation and just out of luck?
                        >
                        > There is a related item in the todo.txt:
                        >
                        > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                        > | pointer in long and seek offset in 64 bit var.
                        >
                        > I wrote some patches to fix this, but they seem to be still unstable.
                        >
                        > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                        > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default

                        Did you make progress on this?
                        Can we also add some tests to verify the fix?

                        --
                        From "know your smileys":
                        :-E Has major dental problems

                        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                        /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                        \\\ an exciting new programming language -- http://www.Zimbu.org ///
                        \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                        --
                        --
                        You received this message from the "vim_use" maillist.
                        Do not top-post! Type your reply below the text you are replying to.
                        For more information, visit http://www.vim.org/maillist.php

                        ---
                        You received this message because you are subscribed to the Google Groups "vim_use" group.
                        To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                        For more options, visit https://groups.google.com/d/optout.
                      • Ken Takata
                        Hi Bram, ... I wrote still unstable , but it seems that it was my mistake. Now I think that the patches are OK. Sometimes Vim (without the patches) freezes
                        Message 11 of 19 , May 9, 2014
                        • 0 Attachment
                          Hi Bram,

                          2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
                          > Ken Takata wrote:
                          >
                          > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                          > > > 'm helping another user open a large, 3Gb, file. The standard windows
                          > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                          > > > after scrolling some amount. For instance, he can't go straight to
                          > > > the end of file.
                          > > >
                          > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                          > > > any settings to modify to make vim more stable with large files, or is
                          > > > there some Windows performance limitation and just out of luck?
                          > >
                          > > There is a related item in the todo.txt:
                          > >
                          > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                          > > | pointer in long and seek offset in 64 bit var.
                          > >
                          > > I wrote some patches to fix this, but they seem to be still unstable.
                          > >
                          > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                          > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                          >
                          > Did you make progress on this?

                          I wrote "still unstable", but it seems that it was my mistake.
                          Now I think that the patches are OK.

                          Sometimes Vim (without the patches) freezes when I open a very big file
                          (about 2 GB) and scroll up and down using scroll bar. After applying the
                          patches, Vim sometimes takes very long time for scrolling up and down, but
                          it wasn't a freeze. (I misunderstood that.)

                          BTW, I found that 32-bit Vim couldn't handle a very big file properly when
                          ":set noswapfile". In my understanding, this is an expected(?) behavior
                          because Vim tries to load the whole file into the memory when 'swapfile' is
                          off, and a 32-bit program can't allocate larger than 2-GiB memory.
                          (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
                          option is specified for 'link'.)


                          > Can we also add some tests to verify the fix?

                          I'm thinking what is the best way to test this.
                          Something like this?

                          " Make sure that a line break is 1 byte.
                          :set ff=unix
                          :set undolevels=-1
                          " Input 99 'A's. The line becomes 100 bytes including a line break.
                          99iA<Esc>
                          yy
                          " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
                          19999999p
                          " Moving around in the file randomly.
                          G
                          10%
                          90%
                          50%
                          gg
                          ...
                          " Edit some lines.
                          ...
                          " Extract some lines and write them to test.out.
                          ...

                          Regards,
                          Ken Takata

                          --
                          --
                          You received this message from the "vim_use" maillist.
                          Do not top-post! Type your reply below the text you are replying to.
                          For more information, visit http://www.vim.org/maillist.php

                          ---
                          You received this message because you are subscribed to the Google Groups "vim_use" group.
                          To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                          For more options, visit https://groups.google.com/d/optout.
                        • Bram Moolenaar
                          ... Although it would be good to test this way, I think it should not be part of make test , since it will probably fail on smaller systems. Perhaps we should
                          Message 12 of 19 , May 10, 2014
                          • 0 Attachment
                            Ken Takata wrote:

                            > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
                            > > Ken Takata wrote:
                            > >
                            > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                            > > > > 'm helping another user open a large, 3Gb, file. The standard windows
                            > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                            > > > > after scrolling some amount. For instance, he can't go straight to
                            > > > > the end of file.
                            > > > >
                            > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                            > > > > any settings to modify to make vim more stable with large files, or is
                            > > > > there some Windows performance limitation and just out of luck?
                            > > >
                            > > > There is a related item in the todo.txt:
                            > > >
                            > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                            > > > | pointer in long and seek offset in 64 bit var.
                            > > >
                            > > > I wrote some patches to fix this, but they seem to be still unstable.
                            > > >
                            > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                            > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                            > >
                            > > Did you make progress on this?
                            >
                            > I wrote "still unstable", but it seems that it was my mistake.
                            > Now I think that the patches are OK.
                            >
                            > Sometimes Vim (without the patches) freezes when I open a very big file
                            > (about 2 GB) and scroll up and down using scroll bar. After applying the
                            > patches, Vim sometimes takes very long time for scrolling up and down, but
                            > it wasn't a freeze. (I misunderstood that.)
                            >
                            > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
                            > ":set noswapfile". In my understanding, this is an expected(?) behavior
                            > because Vim tries to load the whole file into the memory when 'swapfile' is
                            > off, and a 32-bit program can't allocate larger than 2-GiB memory.
                            > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
                            > option is specified for 'link'.)
                            >
                            >
                            > > Can we also add some tests to verify the fix?
                            >
                            > I'm thinking what is the best way to test this.
                            > Something like this?
                            >
                            > " Make sure that a line break is 1 byte.
                            > :set ff=unix
                            > :set undolevels=-1
                            > " Input 99 'A's. The line becomes 100 bytes including a line break.
                            > 99iA<Esc>
                            > yy
                            > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
                            > 19999999p
                            > " Moving around in the file randomly.
                            > G
                            > 10%
                            > 90%
                            > 50%
                            > gg
                            > ...
                            > " Edit some lines.
                            > ...
                            > " Extract some lines and write them to test.out.
                            > ...

                            Although it would be good to test this way, I think it should not be
                            part of "make test", since it will probably fail on smaller systems.
                            Perhaps we should make a "make bigtest" target, for more testing.

                            Generally, I think we need to test that the patch doesn't break anything
                            for normal editing. But perhaps the tests we already have are
                            sufficient for that. If you look at your patch, are there any commands
                            that would not be used by the existing tests?

                            --
                            "The sun oozed over the horizon, shoved aside darkness, crept along the
                            greensward, and, with sickly fingers, pushed through the castle window,
                            revealing the pillaged princess, hand at throat, crown asunder, gaping
                            in frenzied horror at the sated, sodden amphibian lying beside her,
                            disbelieving the magnitude of the frog's deception, screaming madly,
                            "You lied!"
                            - Winner of the Bulwer-Lytton contest (San Jose State University),
                            wherein one writes only the first line of a bad novel

                            /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                            /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                            \\\ an exciting new programming language -- http://www.Zimbu.org ///
                            \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                            --
                            --
                            You received this message from the "vim_use" maillist.
                            Do not top-post! Type your reply below the text you are replying to.
                            For more information, visit http://www.vim.org/maillist.php

                            ---
                            You received this message because you are subscribed to the Google Groups "vim_use" group.
                            To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                            For more options, visit https://groups.google.com/d/optout.
                          • Ken Takata
                            Hi Bram, ... I agree. ... I think that the existing tests cover almost all part, but maybe they doesn t cover the following functions, commands and features: *
                            Message 13 of 19 , May 12, 2014
                            • 0 Attachment
                              Hi Bram,

                              2014/5/10 Sat 20:23:47 UTC+9 Bram Moolenaar wrote:
                              > Ken Takata wrote:
                              >
                              > > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
                              > > > Ken Takata wrote:
                              > > >
                              > > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                              > > > > > 'm helping another user open a large, 3Gb, file. The standard windows
                              > > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                              > > > > > after scrolling some amount. For instance, he can't go straight to
                              > > > > > the end of file.
                              > > > > >
                              > > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                              > > > > > any settings to modify to make vim more stable with large files, or is
                              > > > > > there some Windows performance limitation and just out of luck?
                              > > > >
                              > > > > There is a related item in the todo.txt:
                              > > > >
                              > > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                              > > > > | pointer in long and seek offset in 64 bit var.
                              > > > >
                              > > > > I wrote some patches to fix this, but they seem to be still unstable.
                              > > > >
                              > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                              > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                              > > >
                              > > > Did you make progress on this?
                              > >
                              > > I wrote "still unstable", but it seems that it was my mistake.
                              > > Now I think that the patches are OK.
                              > >
                              > > Sometimes Vim (without the patches) freezes when I open a very big file
                              > > (about 2 GB) and scroll up and down using scroll bar. After applying the
                              > > patches, Vim sometimes takes very long time for scrolling up and down, but
                              > > it wasn't a freeze. (I misunderstood that.)
                              > >
                              > > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
                              > > ":set noswapfile". In my understanding, this is an expected(?) behavior
                              > > because Vim tries to load the whole file into the memory when 'swapfile' is
                              > > off, and a 32-bit program can't allocate larger than 2-GiB memory.
                              > > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
                              > > option is specified for 'link'.)
                              > >
                              > >
                              > > > Can we also add some tests to verify the fix?
                              > >
                              > > I'm thinking what is the best way to test this.
                              > > Something like this?
                              > >
                              > > " Make sure that a line break is 1 byte.
                              > > :set ff=unix
                              > > :set undolevels=-1
                              > > " Input 99 'A's. The line becomes 100 bytes including a line break.
                              > > 99iA<Esc>
                              > > yy
                              > > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
                              > > 19999999p
                              > > " Moving around in the file randomly.
                              > > G
                              > > 10%
                              > > 90%
                              > > 50%
                              > > gg
                              > > ...
                              > > " Edit some lines.
                              > > ...
                              > > " Extract some lines and write them to test.out.
                              > > ...
                              >
                              > Although it would be good to test this way, I think it should not be
                              > part of "make test", since it will probably fail on smaller systems.
                              > Perhaps we should make a "make bigtest" target, for more testing.

                              I agree.


                              > Generally, I think we need to test that the patch doesn't break anything
                              > for normal editing. But perhaps the tests we already have are
                              > sufficient for that. If you look at your patch, are there any commands
                              > that would not be used by the existing tests?

                              I think that the existing tests cover almost all part, but maybe they
                              doesn't cover the following functions, commands and features:
                              * getfsize(), getfperm(), getftime(), getftype()
                              * :checktime
                              * +cscope, +netbeans_intg

                              BTW, I have updated the patch:
                              https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
                              https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
                              * Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
                              * There was no need to use stat_T in os_unix.c. stat_T should be used with
                              mch_stat().

                              Regards,
                              Ken Takata

                              --
                              --
                              You received this message from the "vim_use" maillist.
                              Do not top-post! Type your reply below the text you are replying to.
                              For more information, visit http://www.vim.org/maillist.php

                              ---
                              You received this message because you are subscribed to the Google Groups "vim_use" group.
                              To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                              For more options, visit https://groups.google.com/d/optout.
                            • Bram Moolenaar
                              ... So is the patch now ready to be included, or did you still have a problem to fix? -- hundred-and-one symptoms of being an internet addict: 142. You dream
                              Message 14 of 19 , May 12, 2014
                              • 0 Attachment
                                Ken Takata wrote:

                                > Hi Bram,
                                >
                                > 2014/5/10 Sat 20:23:47 UTC+9 Bram Moolenaar wrote:
                                > > Ken Takata wrote:
                                > >
                                > > > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
                                > > > > Ken Takata wrote:
                                > > > >
                                > > > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                                > > > > > > 'm helping another user open a large, 3Gb, file. The standard windows
                                > > > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                                > > > > > > after scrolling some amount. For instance, he can't go straight to
                                > > > > > > the end of file.
                                > > > > > >
                                > > > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                                > > > > > > any settings to modify to make vim more stable with large files, or is
                                > > > > > > there some Windows performance limitation and just out of luck?
                                > > > > >
                                > > > > > There is a related item in the todo.txt:
                                > > > > >
                                > > > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                                > > > > > | pointer in long and seek offset in 64 bit var.
                                > > > > >
                                > > > > > I wrote some patches to fix this, but they seem to be still unstable.
                                > > > > >
                                > > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                                > > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                                > > > >
                                > > > > Did you make progress on this?
                                > > >
                                > > > I wrote "still unstable", but it seems that it was my mistake.
                                > > > Now I think that the patches are OK.
                                > > >
                                > > > Sometimes Vim (without the patches) freezes when I open a very big file
                                > > > (about 2 GB) and scroll up and down using scroll bar. After applying the
                                > > > patches, Vim sometimes takes very long time for scrolling up and down, but
                                > > > it wasn't a freeze. (I misunderstood that.)
                                > > >
                                > > > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
                                > > > ":set noswapfile". In my understanding, this is an expected(?) behavior
                                > > > because Vim tries to load the whole file into the memory when 'swapfile' is
                                > > > off, and a 32-bit program can't allocate larger than 2-GiB memory.
                                > > > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
                                > > > option is specified for 'link'.)
                                > > >
                                > > >
                                > > > > Can we also add some tests to verify the fix?
                                > > >
                                > > > I'm thinking what is the best way to test this.
                                > > > Something like this?
                                > > >
                                > > > " Make sure that a line break is 1 byte.
                                > > > :set ff=unix
                                > > > :set undolevels=-1
                                > > > " Input 99 'A's. The line becomes 100 bytes including a line break.
                                > > > 99iA<Esc>
                                > > > yy
                                > > > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
                                > > > 19999999p
                                > > > " Moving around in the file randomly.
                                > > > G
                                > > > 10%
                                > > > 90%
                                > > > 50%
                                > > > gg
                                > > > ...
                                > > > " Edit some lines.
                                > > > ...
                                > > > " Extract some lines and write them to test.out.
                                > > > ...
                                > >
                                > > Although it would be good to test this way, I think it should not be
                                > > part of "make test", since it will probably fail on smaller systems.
                                > > Perhaps we should make a "make bigtest" target, for more testing.
                                >
                                > I agree.
                                >
                                >
                                > > Generally, I think we need to test that the patch doesn't break anything
                                > > for normal editing. But perhaps the tests we already have are
                                > > sufficient for that. If you look at your patch, are there any commands
                                > > that would not be used by the existing tests?
                                >
                                > I think that the existing tests cover almost all part, but maybe they
                                > doesn't cover the following functions, commands and features:
                                > * getfsize(), getfperm(), getftime(), getftype()
                                > * :checktime
                                > * +cscope, +netbeans_intg
                                >
                                > BTW, I have updated the patch:
                                > https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
                                > https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
                                > * Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
                                > * There was no need to use stat_T in os_unix.c. stat_T should be used with
                                > mch_stat().

                                So is the patch now ready to be included, or did you still have a
                                problem to fix?

                                --
                                hundred-and-one symptoms of being an internet addict:
                                142. You dream about creating the world's greatest web site.

                                /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                                /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                                \\\ an exciting new programming language -- http://www.Zimbu.org ///
                                \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                                --
                                --
                                You received this message from the "vim_use" maillist.
                                Do not top-post! Type your reply below the text you are replying to.
                                For more information, visit http://www.vim.org/maillist.php

                                ---
                                You received this message because you are subscribed to the Google Groups "vim_use" group.
                                To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                For more options, visit https://groups.google.com/d/optout.
                              • Ken Takata
                                Hi Bram, ... I think it s ready. I don t see any regressions with the patch (at least in my use cases). Updating the tests is another todo item. Regards, Ken
                                Message 15 of 19 , May 12, 2014
                                • 0 Attachment
                                  Hi Bram,

                                  2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
                                  > So is the patch now ready to be included, or did you still have a
                                  > problem to fix?

                                  I think it's ready. I don't see any regressions with the patch (at least in my
                                  use cases).
                                  Updating the tests is another todo item.

                                  Regards,
                                  Ken Takata

                                  --
                                  --
                                  You received this message from the "vim_use" maillist.
                                  Do not top-post! Type your reply below the text you are replying to.
                                  For more information, visit http://www.vim.org/maillist.php

                                  ---
                                  You received this message because you are subscribed to the Google Groups "vim_use" group.
                                  To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                  For more options, visit https://groups.google.com/d/optout.
                                • Bram Moolenaar
                                  ... Well, it s better to have tests before including it. The tests might find a flaw. -- hundred-and-one symptoms of being an internet addict: 149. You find
                                  Message 16 of 19 , May 13, 2014
                                  • 0 Attachment
                                    Ken Takata wrote:

                                    > 2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
                                    > > So is the patch now ready to be included, or did you still have a
                                    > > problem to fix?
                                    >
                                    > I think it's ready. I don't see any regressions with the patch (at least in my
                                    > use cases).
                                    > Updating the tests is another todo item.

                                    Well, it's better to have tests before including it. The tests might
                                    find a flaw.

                                    --
                                    hundred-and-one symptoms of being an internet addict:
                                    149. You find your computer sexier than your girlfriend

                                    /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                                    /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                                    \\\ an exciting new programming language -- http://www.Zimbu.org ///
                                    \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                                    --
                                    --
                                    You received this message from the "vim_use" maillist.
                                    Do not top-post! Type your reply below the text you are replying to.
                                    For more information, visit http://www.vim.org/maillist.php

                                    ---
                                    You received this message because you are subscribed to the Google Groups "vim_use" group.
                                    To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                    For more options, visit https://groups.google.com/d/optout.
                                  • Ken Takata
                                    Hi Bram, ... OK, I wrote very simple tests for getfsize(), getfperm(), getftime(), getftype() and :checktime.
                                    Message 17 of 19 , May 14, 2014
                                    • 0 Attachment
                                      Hi Bram,

                                      2014/5/13 Tue 20:12:59 UTC+9 Bram Moolenaar wrote:
                                      > Ken Takata wrote:
                                      >
                                      > > 2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
                                      > > > So is the patch now ready to be included, or did you still have a
                                      > > > problem to fix?
                                      > >
                                      > > I think it's ready. I don't see any regressions with the patch (at least in my
                                      > > use cases).
                                      > > Updating the tests is another todo item.
                                      >
                                      > Well, it's better to have tests before including it. The tests might
                                      > find a flaw.

                                      OK, I wrote very simple tests for getfsize(), getfperm(), getftime(),
                                      getftype() and :checktime.
                                      https://bitbucket.org/k_takata/vim-ktakata-mq/src/09e276a96d721ded06d8a140e730638ffdfb18ca/add-stat-test.patch

                                      Regards,
                                      Ken Takata

                                      --
                                      --
                                      You received this message from the "vim_use" maillist.
                                      Do not top-post! Type your reply below the text you are replying to.
                                      For more information, visit http://www.vim.org/maillist.php

                                      ---
                                      You received this message because you are subscribed to the Google Groups "vim_use" group.
                                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                      For more options, visit https://groups.google.com/d/optout.
                                    • Ken Takata
                                      Hi Bram, ... I have updated the tests for 7.4.315. (Just renamed from test107 to test108.)
                                      Message 18 of 19 , May 29, 2014
                                      • 0 Attachment
                                        Hi Bram,

                                        2014/5/14 Wed 23:19:38 UTC+9 Ken Takata wrote:
                                        > Hi Bram,
                                        >
                                        > 2014/5/13 Tue 20:12:59 UTC+9 Bram Moolenaar wrote:
                                        > > Ken Takata wrote:
                                        > >
                                        > > > 2014/5/13 Tue 3:40:37 UTC+9 Bram Moolenaar wrote:
                                        > > > > So is the patch now ready to be included, or did you still have a
                                        > > > > problem to fix?
                                        > > >
                                        > > > I think it's ready. I don't see any regressions with the patch (at least in my
                                        > > > use cases).
                                        > > > Updating the tests is another todo item.
                                        > >
                                        > > Well, it's better to have tests before including it. The tests might
                                        > > find a flaw.
                                        >
                                        > OK, I wrote very simple tests for getfsize(), getfperm(), getftime(),
                                        > getftype() and :checktime.
                                        > https://bitbucket.org/k_takata/vim-ktakata-mq/src/09e276a96d721ded06d8a140e730638ffdfb18ca/add-stat-test.patch

                                        I have updated the tests for 7.4.315. (Just renamed from test107 to test108.)
                                        https://bitbucket.org/k_takata/vim-ktakata-mq/src/tip/add-stat-test.patch

                                        Regards,
                                        Ken Takata

                                        --
                                        --
                                        You received this message from the "vim_use" maillist.
                                        Do not top-post! Type your reply below the text you are replying to.
                                        For more information, visit http://www.vim.org/maillist.php

                                        ---
                                        You received this message because you are subscribed to the Google Groups "vim_use" group.
                                        To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                        For more options, visit https://groups.google.com/d/optout.
                                      • Ken Takata
                                        Hi Bram, ... After 7.4.399, support-largefiles-on-windows.patch couldn t be applied because of conflicts. So I have updated the patch. The latest patches are:
                                        Message 19 of 19 , Aug 10, 2014
                                        • 0 Attachment
                                          Hi Bram,

                                          2014/5/12 Mon 22:17:05 UTC+9 Ken Takata wrote:
                                          > Hi Bram,
                                          >
                                          > 2014/5/10 Sat 20:23:47 UTC+9 Bram Moolenaar wrote:
                                          > > Ken Takata wrote:
                                          > >
                                          > > > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
                                          > > > > Ken Takata wrote:
                                          > > > >
                                          > > > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
                                          > > > > > > 'm helping another user open a large, 3Gb, file. The standard windows
                                          > > > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
                                          > > > > > > after scrolling some amount. For instance, he can't go straight to
                                          > > > > > > the end of file.
                                          > > > > > >
                                          > > > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
                                          > > > > > > any settings to modify to make vim more stable with large files, or is
                                          > > > > > > there some Windows performance limitation and just out of luck?
                                          > > > > >
                                          > > > > > There is a related item in the todo.txt:
                                          > > > > >
                                          > > > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
                                          > > > > > | pointer in long and seek offset in 64 bit var.
                                          > > > > >
                                          > > > > > I wrote some patches to fix this, but they seem to be still unstable.
                                          > > > > >
                                          > > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
                                          > > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
                                          > > > >
                                          > > > > Did you make progress on this?
                                          > > >
                                          > > > I wrote "still unstable", but it seems that it was my mistake.
                                          > > > Now I think that the patches are OK.
                                          > > >
                                          > > > Sometimes Vim (without the patches) freezes when I open a very big file
                                          > > > (about 2 GB) and scroll up and down using scroll bar. After applying the
                                          > > > patches, Vim sometimes takes very long time for scrolling up and down, but
                                          > > > it wasn't a freeze. (I misunderstood that.)
                                          > > >
                                          > > > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
                                          > > > ":set noswapfile". In my understanding, this is an expected(?) behavior
                                          > > > because Vim tries to load the whole file into the memory when 'swapfile' is
                                          > > > off, and a 32-bit program can't allocate larger than 2-GiB memory.
                                          > > > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
                                          > > > option is specified for 'link'.)
                                          > > >
                                          > > >
                                          > > > > Can we also add some tests to verify the fix?
                                          > > >
                                          > > > I'm thinking what is the best way to test this.
                                          > > > Something like this?
                                          > > >
                                          > > > " Make sure that a line break is 1 byte.
                                          > > > :set ff=unix
                                          > > > :set undolevels=-1
                                          > > > " Input 99 'A's. The line becomes 100 bytes including a line break.
                                          > > > 99iA<Esc>
                                          > > > yy
                                          > > > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
                                          > > > 19999999p
                                          > > > " Moving around in the file randomly.
                                          > > > G
                                          > > > 10%
                                          > > > 90%
                                          > > > 50%
                                          > > > gg
                                          > > > ...
                                          > > > " Edit some lines.
                                          > > > ...
                                          > > > " Extract some lines and write them to test.out.
                                          > > > ...
                                          > >
                                          > > Although it would be good to test this way, I think it should not be
                                          > > part of "make test", since it will probably fail on smaller systems.
                                          > > Perhaps we should make a "make bigtest" target, for more testing.
                                          >
                                          > I agree.
                                          >
                                          >
                                          > > Generally, I think we need to test that the patch doesn't break anything
                                          > > for normal editing. But perhaps the tests we already have are
                                          > > sufficient for that. If you look at your patch, are there any commands
                                          > > that would not be used by the existing tests?
                                          >
                                          > I think that the existing tests cover almost all part, but maybe they
                                          > doesn't cover the following functions, commands and features:
                                          > * getfsize(), getfperm(), getftime(), getftype()
                                          > * :checktime
                                          > * +cscope, +netbeans_intg
                                          >
                                          > BTW, I have updated the patch:
                                          > https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
                                          > https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
                                          > * Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
                                          > * There was no need to use stat_T in os_unix.c. stat_T should be used with
                                          > mch_stat().

                                          After 7.4.399, support-largefiles-on-windows.patch couldn't be applied because of
                                          conflicts. So I have updated the patch.
                                          The latest patches are:

                                          https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/support-largefiles-on-windows.patch?at=default
                                          https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/use-stat_T.patch?at=default
                                          https://bitbucket.org/k_takata/vim-ktakata-mq/src/be6f89cd8a129025d2a456eb60b946265d90dffe/add-stat-test.patch?at=default

                                          Regards,
                                          Ken Takata

                                          --
                                          --
                                          You received this message from the "vim_use" maillist.
                                          Do not top-post! Type your reply below the text you are replying to.
                                          For more information, visit http://www.vim.org/maillist.php

                                          ---
                                          You received this message because you are subscribed to the Google Groups "vim_use" group.
                                          To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                                          For more options, visit https://groups.google.com/d/optout.
                                        Your message has been successfully submitted and would be delivered to recipients shortly.