Loading ...
Sorry, an error occurred while loading the content.
 

vim peculiarity with clearcase

Expand Messages
  • Paul_A
    I ve recently run across an odd situation in which vim essentially becomes unusable, but I d at least like to find a workaround. I started seeing this when
    Message 1 of 6 , Jun 2, 2000
      I've recently run across an odd situation in which vim essentially
      becomes unusable, but I'd at least like to find a workaround.

      I started seeing this when running vim on a list of about 40 files.
      I'm doing something like "vim `cat filelist`", where the files
      named are all on a clearcase filesystem. I don't know much about
      these filesystems, but they seem to be kind of a proprietary NFS.
      What happens is that vim takes roughly 10 minutes before it's ready
      to accept input. The behaviour is about the same with both 5.3 and
      5.6 with some patches, and with the GUI as well; the OS is Solaris 2.6.

      When paring down the list of files, I notice that each additional one
      takes from 12 to 15 seconds, so that would explain the 10 minute total.
      It would appear vim opens all the files given on a command line before
      the first on can be editted -- is there a good reason for that, or is
      that even true? Is this some sort of initialization of the buffer list?

      I realize the slowness could possibly be related to the network here,
      but vim is the only program where I've seen anything like this delay.
      Normal NFS is 100 time as fast, but I admit that's on a different server.
      In addition, I've tried truss'ing the process and seen that it appears
      very busy in some kind of a loop until it does eventually finish.

      Does anyone have any idea what could be going on here?

      --
      Paul Ackersviller
    • Bram Moolenaar
      ... The problem is with clearcase. Vim tries to get the full pathname of each file. This is required to check if it s the same file as edited before from
      Message 2 of 6 , Jun 3, 2000
        Paul Ackersviller wrote:

        > I've recently run across an odd situation in which vim essentially
        > becomes unusable, but I'd at least like to find a workaround.
        >
        > I started seeing this when running vim on a list of about 40 files.
        > I'm doing something like "vim `cat filelist`", where the files
        > named are all on a clearcase filesystem. I don't know much about
        > these filesystems, but they seem to be kind of a proprietary NFS.
        > What happens is that vim takes roughly 10 minutes before it's ready
        > to accept input. The behaviour is about the same with both 5.3 and
        > 5.6 with some patches, and with the GUI as well; the OS is Solaris 2.6.

        The problem is with clearcase. Vim tries to get the full pathname of each
        file. This is required to check if it's the same file as edited before from
        another directory. This is nothing special, but the files in the clearcase
        system just respond very slow.

        Try doing "pwd" in a clearcase directory. I suspect that's slow too.

        > Does anyone have any idea what could be going on here?

        You could use "truss" to see which system calls cause the delay. If I
        remember correctly, somebody said it was the "getdents()" call.

        --
        Wi n0t trei a h0liday in Sweden thi yer?
        "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

        /-/-- Bram Moolenaar --- Bram@... --- http://www.moolenaar.net --\-\
        \-\-- Vim: http://www.vim.org ---- ICCF Holland: http://www.vim.org/iccf --/-/
      • Gregory Margo
        ... Why does vim need to find the full pathname of a file before it is actually being edited? I ve experienced this slowness in vim whenever I attempt to edit
        Message 3 of 6 , Jun 3, 2000
          On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
          >
          > Paul Ackersviller wrote:
          >
          > > I've recently run across an odd situation in which vim essentially
          > > becomes unusable, but I'd at least like to find a workaround.
          > >
          > > I started seeing this when running vim on a list of about 40 files.
          > > I'm doing something like "vim `cat filelist`", where the files
          > > named are all on a clearcase filesystem. I don't know much about
          > > these filesystems, but they seem to be kind of a proprietary NFS.
          > > What happens is that vim takes roughly 10 minutes before it's ready
          > > to accept input. The behaviour is about the same with both 5.3 and
          > > 5.6 with some patches, and with the GUI as well; the OS is Solaris 2.6.
          >
          > The problem is with clearcase. Vim tries to get the full pathname of each
          > file. This is required to check if it's the same file as edited before from
          > another directory. This is nothing special, but the files in the clearcase
          > system just respond very slow.
          >
          > Try doing "pwd" in a clearcase directory. I suspect that's slow too.
          >
          > > Does anyone have any idea what could be going on here?
          >
          > You could use "truss" to see which system calls cause the delay. If I
          > remember correctly, somebody said it was the "getdents()" call.

          Why does vim need to find the full pathname of a file before it is
          actually being edited? I've experienced this slowness in vim whenever I
          attempt to edit a long list of files, whether in Clearcase or not.

          (BTW, Paul's Clearcase seems abnormally slow - perhaps the sysadmin
          needs to check the size of the vob - too many files in the vob makes
          access slow. Or delete and recreate the view - gudge builds up in a view.)

          --
          ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
          Gregory H. Margo
          Home: gmargo@... gmargo@... ghmargo@...
          Work: greg_margo@...
        • Paul_A
          ... Yes, I agree the problem has more to do with clearcase. But couldn t vim operate more efficiently by working with inode numbers, which are unique on a
          Message 4 of 6 , Jun 5, 2000
            On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
            > The problem is with clearcase. Vim tries to get the full pathname of each
            > file. This is required to check if it's the same file as edited before from
            > another directory. This is nothing special, but the files in the clearcase
            > system just respond very slow.

            Yes, I agree the problem has more to do with clearcase. But couldn't vim
            operate more efficiently by working with inode numbers, which are unique
            on a given filesystem (not to mention just opening one file at a time)?

            > Try doing "pwd" in a clearcase directory. I suspect that's slow too.

            Actually it's not, the delay is due to the names of the files. Someone
            had given me a list of names showing the path, followed by `@@' and the
            exact branch. Response time is normal without the branch specification.
            Nevertheless, I found it disconcerting that the full path lookup took so
            long in vim where other program exhibited nothing unusual.

            > You could use "truss" to see which system calls cause the delay. If I
            > remember correctly, somebody said it was the "getdents()" call.

            It appears to be mostly waiting a bit after some fstat64 calls, but the
            output is fast enough that it's hard to tell with the naked eye. Since
            it's nothing unintentional in vim, I won't do any more detailed profiling.

            --
            Paul Ackersviller
          • Bram Moolenaar
            ... It looks like the time is spend in the stat() call, which is the call required to get the inode. Inodes are already used to compare files. Vim also
            Message 5 of 6 , Jun 6, 2000
              Paul Ackersviller wrote:

              > On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
              > > The problem is with clearcase. Vim tries to get the full pathname of each
              > > file. This is required to check if it's the same file as edited before
              > > from another directory. This is nothing special, but the files in the
              > > clearcase system just respond very slow.
              >
              > Yes, I agree the problem has more to do with clearcase. But couldn't vim
              > operate more efficiently by working with inode numbers, which are unique
              > on a given filesystem (not to mention just opening one file at a time)?

              It looks like the time is spend in the stat() call, which is the call required
              to get the inode. Inodes are already used to compare files. Vim also stores
              the absolute path name, because that is what really identifies a file for the
              user (a file could be deleted and created again, giving it a new inode number,
              but it's still the same file from the users point of view). This mostly
              matters for the viminfo feature.

              > > Try doing "pwd" in a clearcase directory. I suspect that's slow too.
              >
              > Actually it's not, the delay is due to the names of the files. Someone
              > had given me a list of names showing the path, followed by `@@' and the
              > exact branch. Response time is normal without the branch specification.
              > Nevertheless, I found it disconcerting that the full path lookup took so
              > long in vim where other program exhibited nothing unusual.

              Doing a "truss" on Vim it looks like there are three stat() calls for each
              file. I never paid much attention to this, assuming that stat() is reasonably
              fast. The first two stat calls might be reduced to one: The first is used to
              check if the same file was already encountered, and if isn't it is added to
              the list of buffers, which requires another stat() call. The third call is
              for the Buffers menu, you could disable the Buffers menu to avoid this one.

              > > You could use "truss" to see which system calls cause the delay. If I
              > > remember correctly, somebody said it was the "getdents()" call.
              >
              > It appears to be mostly waiting a bit after some fstat64 calls, but the
              > output is fast enough that it's hard to tell with the naked eye. Since
              > it's nothing unintentional in vim, I won't do any more detailed profiling.

              Profiling would still be useful, because it often shows surprising things.
              For example, it might be that only the first stat() call for a file is slow,
              and following ones are fast (because of caching). Reducing the number of
              stat() calls for a file won't help then. You could also use ktrace/kdump to
              see how much time is spend in each system call.

              You could try out the patch below. It halves the number of stat() calls for
              each file added to the buffer list. Let me know if this significantly reduces
              the time to start Vim. For me it's a matter of a few percent (starting with
              about 1500 files takes about three seconds). It could matter a lot more when
              using a network or clearcase.


              *** buffer.c.orig Tue Jun 6 10:50:19 2000
              --- buffer.c Tue Jun 6 11:02:43 2000
              ***************
              *** 31,36 ****
              --- 31,37 ----
              static char_u *buflist_match_try __ARGS((vim_regexp *prog, char_u *name));
              static void buflist_setfpos __ARGS((BUF *, linenr_t, colnr_t));
              #ifdef UNIX
              + static BUF *buflist_findname_stat __ARGS((char_u *ffname, struct stat *st));
              static int otherfile_buf __ARGS((BUF *buf, char_u *ffname, struct stat *stp));
              static void buf_setino __ARGS((BUF *buf));
              static int buf_same_ino __ARGS((BUF *buf, struct stat *stp));
              ***************
              *** 802,814 ****
              int use_curbuf;
              {
              BUF *buf;

              fname_expand(&ffname, &sfname); /* will allocate ffname */

              /*
              * If file name already exists in the list, update the entry.
              */
              ! if (ffname != NULL && (buf = buflist_findname(ffname)) != NULL)
              {
              vim_free(ffname);
              if (lnum != 0)
              --- 803,829 ----
              int use_curbuf;
              {
              BUF *buf;
              + #ifdef UNIX
              + struct stat st;
              + #endif

              fname_expand(&ffname, &sfname); /* will allocate ffname */

              + #ifdef UNIX
              + if (sfname == NULL || mch_stat((char *)sfname, &st) < 0)
              + st.st_dev = (unsigned)-1;
              + #endif
              +
              /*
              * If file name already exists in the list, update the entry.
              */
              ! if (ffname != NULL && (buf =
              ! #ifdef UNIX
              ! buflist_findname_stat(ffname, &st)
              ! #else
              ! buflist_findname(ffname)
              ! #endif
              ! ) != NULL)
              {
              vim_free(ffname);
              if (lnum != 0)
              ***************
              *** 917,923 ****

              buf->b_fname = buf->b_sfname;
              #ifdef UNIX
              ! buf_setino(buf);
              #endif
              buf->b_u_synced = TRUE;
              buf->b_flags = BF_CHECK_RO | BF_NEVERLOADED;
              --- 932,944 ----

              buf->b_fname = buf->b_sfname;
              #ifdef UNIX
              ! if (st.st_dev == (unsigned)-1)
              ! buf->b_dev = -1;
              ! else
              ! {
              ! buf->b_dev = st.st_dev;
              ! buf->b_ino = st.st_ino;
              ! }
              #endif
              buf->b_u_synced = TRUE;
              buf->b_flags = BF_CHECK_RO | BF_NEVERLOADED;
              ***************
              *** 1093,1110 ****
              buflist_findname(ffname)
              char_u *ffname;
              {
              - BUF *buf;
              #ifdef UNIX
              struct stat st;

              if (mch_stat((char *)ffname, &st) < 0)
              st.st_dev = (unsigned)-1;
              #endif

              for (buf = firstbuf; buf != NULL; buf = buf->b_next)
              if (!otherfile_buf(buf, ffname
              #ifdef UNIX
              ! , &st
              #endif
              ))
              return buf;
              --- 1114,1143 ----
              buflist_findname(ffname)
              char_u *ffname;
              {
              #ifdef UNIX
              struct stat st;

              if (mch_stat((char *)ffname, &st) < 0)
              st.st_dev = (unsigned)-1;
              + return buflist_findname_stat(ffname, &st);
              + }
              +
              + /*
              + * Same as buflist_findname(), but pass the stat structure to avoid getting it
              + * twice for the same file.
              + */
              + static BUF *
              + buflist_findname_stat(ffname, stp)
              + char_u *ffname;
              + struct stat *stp;
              + {
              #endif
              + BUF *buf;

              for (buf = firstbuf; buf != NULL; buf = buf->b_next)
              if (!otherfile_buf(buf, ffname
              #ifdef UNIX
              ! , stp
              #endif
              ))
              return buf;
              ***************
              *** 1536,1542 ****
              char_u *ffname, *sfname;
              int message;
              {
              ! BUF *buf;

              if (ffname == NULL || *ffname == NUL)
              {
              --- 1569,1578 ----
              char_u *ffname, *sfname;
              int message;
              {
              ! BUF *buf;
              ! #ifdef UNIX
              ! struct stat st;
              ! #endif

              if (ffname == NULL || *ffname == NUL)
              {
              ***************
              *** 1544,1549 ****
              --- 1580,1588 ----
              vim_free(curbuf->b_sfname);
              curbuf->b_ffname = NULL;
              curbuf->b_sfname = NULL;
              + #ifdef UNIX
              + st.st_dev = (unsigned)-1;
              + #endif
              }
              else
              {
              ***************
              *** 1562,1568 ****
              --- 1601,1613 ----
              * - if the buffer is loaded, fail
              * - if the buffer is not loaded, delete it from the list
              */
              + #ifdef UNIX
              + if (mch_stat((char *)ffname, &st) < 0)
              + st.st_dev = (unsigned)-1;
              + buf = buflist_findname_stat(ffname, &st);
              + #else
              buf = buflist_findname(ffname);
              + #endif
              if (buf != NULL && buf != curbuf)
              {
              if (buf->b_ml.ml_mfp != NULL) /* it's loaded, fail */
              ***************
              *** 1588,1594 ****
              }
              curbuf->b_fname = curbuf->b_sfname;
              #ifdef UNIX
              ! buf_setino(curbuf);
              #endif

              #ifndef SHORT_FNAME
              --- 1633,1645 ----
              }
              curbuf->b_fname = curbuf->b_sfname;
              #ifdef UNIX
              ! if (st.st_dev == (unsigned)-1)
              ! curbuf->b_dev = -1;
              ! else
              ! {
              ! curbuf->b_dev = st.st_dev;
              ! curbuf->b_ino = st.st_ino;
              ! }
              #endif

              #ifndef SHORT_FNAME

              --
              OLD WOMAN: King of the WHO?
              ARTHUR: The Britons.
              OLD WOMAN: Who are the Britons?
              "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

              /-/-- Bram Moolenaar --- Bram@... --- http://www.moolenaar.net --\-\
              \-\-- Vim: http://www.vim.org ---- ICCF Holland: http://www.vim.org/iccf --/-/
            • Paul_A
              ... Thanks for that explanation, I hadn t considered the possibility of the file being deleted in the interim. ... I haven t tried any profiling yet, but have
              Message 6 of 6 , Jun 6, 2000
                On Tue, Jun 06, 2000 at 12:32:06PM +0200, Bram Moolenaar wrote:
                > It looks like the time is spend in the stat() call, which is the call required
                > to get the inode. Inodes are already used to compare files. Vim also stores
                > the absolute path name, because that is what really identifies a file for the
                > user (a file could be deleted and created again, giving it a new inode number,
                > but it's still the same file from the users point of view). This mostly
                > matters for the viminfo feature.

                Thanks for that explanation, I hadn't considered the possibility of the file
                being deleted in the interim.

                > Profiling would still be useful, because it often shows surprising things.
                > For example, it might be that only the first stat() call for a file is slow,
                > and following ones are fast (because of caching). Reducing the number of
                > stat() calls for a file won't help then. You could also use ktrace/kdump to
                > see how much time is spend in each system call.

                I haven't tried any profiling yet, but have given your patch a whirl.
                It looks like your suspicion about the cache effect was correct, eliminating
                the extra stat of each file doesn't seem to help. Strangely enough, it
                mostly measured slower on several repetitions, but it was hardly scientific
                and the numbers were up and down a lot.

                > You could try out the patch below. It halves the number of stat() calls for
                > each file added to the buffer list. Let me know if this significantly reduces
                > the time to start Vim. For me it's a matter of a few percent (starting with
                > about 1500 files takes about three seconds). It could matter a lot more when
                > using a network or clearcase.

                Thanks Bram, I'll let you know if profiling uncovers anything interesting.

                --
                Paul Ackersviller
              Your message has been successfully submitted and would be delivered to recipients shortly.