Loading ...
Sorry, an error occurred while loading the content.

Re: vim peculiarity with clearcase

Expand Messages
  • Bram Moolenaar
    ... The problem is with clearcase. Vim tries to get the full pathname of each file. This is required to check if it s the same file as edited before from
    Message 1 of 6 , Jun 3, 2000
    • 0 Attachment
      Paul Ackersviller wrote:

      > I've recently run across an odd situation in which vim essentially
      > becomes unusable, but I'd at least like to find a workaround.
      >
      > I started seeing this when running vim on a list of about 40 files.
      > I'm doing something like "vim `cat filelist`", where the files
      > named are all on a clearcase filesystem. I don't know much about
      > these filesystems, but they seem to be kind of a proprietary NFS.
      > What happens is that vim takes roughly 10 minutes before it's ready
      > to accept input. The behaviour is about the same with both 5.3 and
      > 5.6 with some patches, and with the GUI as well; the OS is Solaris 2.6.

      The problem is with clearcase. Vim tries to get the full pathname of each
      file. This is required to check if it's the same file as edited before from
      another directory. This is nothing special, but the files in the clearcase
      system just respond very slow.

      Try doing "pwd" in a clearcase directory. I suspect that's slow too.

      > Does anyone have any idea what could be going on here?

      You could use "truss" to see which system calls cause the delay. If I
      remember correctly, somebody said it was the "getdents()" call.

      --
      Wi n0t trei a h0liday in Sweden thi yer?
      "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

      /-/-- Bram Moolenaar --- Bram@... --- http://www.moolenaar.net --\-\
      \-\-- Vim: http://www.vim.org ---- ICCF Holland: http://www.vim.org/iccf --/-/
    • Gregory Margo
      ... Why does vim need to find the full pathname of a file before it is actually being edited? I ve experienced this slowness in vim whenever I attempt to edit
      Message 2 of 6 , Jun 3, 2000
      • 0 Attachment
        On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
        >
        > Paul Ackersviller wrote:
        >
        > > I've recently run across an odd situation in which vim essentially
        > > becomes unusable, but I'd at least like to find a workaround.
        > >
        > > I started seeing this when running vim on a list of about 40 files.
        > > I'm doing something like "vim `cat filelist`", where the files
        > > named are all on a clearcase filesystem. I don't know much about
        > > these filesystems, but they seem to be kind of a proprietary NFS.
        > > What happens is that vim takes roughly 10 minutes before it's ready
        > > to accept input. The behaviour is about the same with both 5.3 and
        > > 5.6 with some patches, and with the GUI as well; the OS is Solaris 2.6.
        >
        > The problem is with clearcase. Vim tries to get the full pathname of each
        > file. This is required to check if it's the same file as edited before from
        > another directory. This is nothing special, but the files in the clearcase
        > system just respond very slow.
        >
        > Try doing "pwd" in a clearcase directory. I suspect that's slow too.
        >
        > > Does anyone have any idea what could be going on here?
        >
        > You could use "truss" to see which system calls cause the delay. If I
        > remember correctly, somebody said it was the "getdents()" call.

        Why does vim need to find the full pathname of a file before it is
        actually being edited? I've experienced this slowness in vim whenever I
        attempt to edit a long list of files, whether in Clearcase or not.

        (BTW, Paul's Clearcase seems abnormally slow - perhaps the sysadmin
        needs to check the size of the vob - too many files in the vob makes
        access slow. Or delete and recreate the view - gudge builds up in a view.)

        --
        ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
        Gregory H. Margo
        Home: gmargo@... gmargo@... ghmargo@...
        Work: greg_margo@...
      • Paul_A
        ... Yes, I agree the problem has more to do with clearcase. But couldn t vim operate more efficiently by working with inode numbers, which are unique on a
        Message 3 of 6 , Jun 5, 2000
        • 0 Attachment
          On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
          > The problem is with clearcase. Vim tries to get the full pathname of each
          > file. This is required to check if it's the same file as edited before from
          > another directory. This is nothing special, but the files in the clearcase
          > system just respond very slow.

          Yes, I agree the problem has more to do with clearcase. But couldn't vim
          operate more efficiently by working with inode numbers, which are unique
          on a given filesystem (not to mention just opening one file at a time)?

          > Try doing "pwd" in a clearcase directory. I suspect that's slow too.

          Actually it's not, the delay is due to the names of the files. Someone
          had given me a list of names showing the path, followed by `@@' and the
          exact branch. Response time is normal without the branch specification.
          Nevertheless, I found it disconcerting that the full path lookup took so
          long in vim where other program exhibited nothing unusual.

          > You could use "truss" to see which system calls cause the delay. If I
          > remember correctly, somebody said it was the "getdents()" call.

          It appears to be mostly waiting a bit after some fstat64 calls, but the
          output is fast enough that it's hard to tell with the naked eye. Since
          it's nothing unintentional in vim, I won't do any more detailed profiling.

          --
          Paul Ackersviller
        • Bram Moolenaar
          ... It looks like the time is spend in the stat() call, which is the call required to get the inode. Inodes are already used to compare files. Vim also
          Message 4 of 6 , Jun 6, 2000
          • 0 Attachment
            Paul Ackersviller wrote:

            > On Sat, Jun 03, 2000 at 12:59:51PM +0200, Bram Moolenaar wrote:
            > > The problem is with clearcase. Vim tries to get the full pathname of each
            > > file. This is required to check if it's the same file as edited before
            > > from another directory. This is nothing special, but the files in the
            > > clearcase system just respond very slow.
            >
            > Yes, I agree the problem has more to do with clearcase. But couldn't vim
            > operate more efficiently by working with inode numbers, which are unique
            > on a given filesystem (not to mention just opening one file at a time)?

            It looks like the time is spend in the stat() call, which is the call required
            to get the inode. Inodes are already used to compare files. Vim also stores
            the absolute path name, because that is what really identifies a file for the
            user (a file could be deleted and created again, giving it a new inode number,
            but it's still the same file from the users point of view). This mostly
            matters for the viminfo feature.

            > > Try doing "pwd" in a clearcase directory. I suspect that's slow too.
            >
            > Actually it's not, the delay is due to the names of the files. Someone
            > had given me a list of names showing the path, followed by `@@' and the
            > exact branch. Response time is normal without the branch specification.
            > Nevertheless, I found it disconcerting that the full path lookup took so
            > long in vim where other program exhibited nothing unusual.

            Doing a "truss" on Vim it looks like there are three stat() calls for each
            file. I never paid much attention to this, assuming that stat() is reasonably
            fast. The first two stat calls might be reduced to one: The first is used to
            check if the same file was already encountered, and if isn't it is added to
            the list of buffers, which requires another stat() call. The third call is
            for the Buffers menu, you could disable the Buffers menu to avoid this one.

            > > You could use "truss" to see which system calls cause the delay. If I
            > > remember correctly, somebody said it was the "getdents()" call.
            >
            > It appears to be mostly waiting a bit after some fstat64 calls, but the
            > output is fast enough that it's hard to tell with the naked eye. Since
            > it's nothing unintentional in vim, I won't do any more detailed profiling.

            Profiling would still be useful, because it often shows surprising things.
            For example, it might be that only the first stat() call for a file is slow,
            and following ones are fast (because of caching). Reducing the number of
            stat() calls for a file won't help then. You could also use ktrace/kdump to
            see how much time is spend in each system call.

            You could try out the patch below. It halves the number of stat() calls for
            each file added to the buffer list. Let me know if this significantly reduces
            the time to start Vim. For me it's a matter of a few percent (starting with
            about 1500 files takes about three seconds). It could matter a lot more when
            using a network or clearcase.


            *** buffer.c.orig Tue Jun 6 10:50:19 2000
            --- buffer.c Tue Jun 6 11:02:43 2000
            ***************
            *** 31,36 ****
            --- 31,37 ----
            static char_u *buflist_match_try __ARGS((vim_regexp *prog, char_u *name));
            static void buflist_setfpos __ARGS((BUF *, linenr_t, colnr_t));
            #ifdef UNIX
            + static BUF *buflist_findname_stat __ARGS((char_u *ffname, struct stat *st));
            static int otherfile_buf __ARGS((BUF *buf, char_u *ffname, struct stat *stp));
            static void buf_setino __ARGS((BUF *buf));
            static int buf_same_ino __ARGS((BUF *buf, struct stat *stp));
            ***************
            *** 802,814 ****
            int use_curbuf;
            {
            BUF *buf;

            fname_expand(&ffname, &sfname); /* will allocate ffname */

            /*
            * If file name already exists in the list, update the entry.
            */
            ! if (ffname != NULL && (buf = buflist_findname(ffname)) != NULL)
            {
            vim_free(ffname);
            if (lnum != 0)
            --- 803,829 ----
            int use_curbuf;
            {
            BUF *buf;
            + #ifdef UNIX
            + struct stat st;
            + #endif

            fname_expand(&ffname, &sfname); /* will allocate ffname */

            + #ifdef UNIX
            + if (sfname == NULL || mch_stat((char *)sfname, &st) < 0)
            + st.st_dev = (unsigned)-1;
            + #endif
            +
            /*
            * If file name already exists in the list, update the entry.
            */
            ! if (ffname != NULL && (buf =
            ! #ifdef UNIX
            ! buflist_findname_stat(ffname, &st)
            ! #else
            ! buflist_findname(ffname)
            ! #endif
            ! ) != NULL)
            {
            vim_free(ffname);
            if (lnum != 0)
            ***************
            *** 917,923 ****

            buf->b_fname = buf->b_sfname;
            #ifdef UNIX
            ! buf_setino(buf);
            #endif
            buf->b_u_synced = TRUE;
            buf->b_flags = BF_CHECK_RO | BF_NEVERLOADED;
            --- 932,944 ----

            buf->b_fname = buf->b_sfname;
            #ifdef UNIX
            ! if (st.st_dev == (unsigned)-1)
            ! buf->b_dev = -1;
            ! else
            ! {
            ! buf->b_dev = st.st_dev;
            ! buf->b_ino = st.st_ino;
            ! }
            #endif
            buf->b_u_synced = TRUE;
            buf->b_flags = BF_CHECK_RO | BF_NEVERLOADED;
            ***************
            *** 1093,1110 ****
            buflist_findname(ffname)
            char_u *ffname;
            {
            - BUF *buf;
            #ifdef UNIX
            struct stat st;

            if (mch_stat((char *)ffname, &st) < 0)
            st.st_dev = (unsigned)-1;
            #endif

            for (buf = firstbuf; buf != NULL; buf = buf->b_next)
            if (!otherfile_buf(buf, ffname
            #ifdef UNIX
            ! , &st
            #endif
            ))
            return buf;
            --- 1114,1143 ----
            buflist_findname(ffname)
            char_u *ffname;
            {
            #ifdef UNIX
            struct stat st;

            if (mch_stat((char *)ffname, &st) < 0)
            st.st_dev = (unsigned)-1;
            + return buflist_findname_stat(ffname, &st);
            + }
            +
            + /*
            + * Same as buflist_findname(), but pass the stat structure to avoid getting it
            + * twice for the same file.
            + */
            + static BUF *
            + buflist_findname_stat(ffname, stp)
            + char_u *ffname;
            + struct stat *stp;
            + {
            #endif
            + BUF *buf;

            for (buf = firstbuf; buf != NULL; buf = buf->b_next)
            if (!otherfile_buf(buf, ffname
            #ifdef UNIX
            ! , stp
            #endif
            ))
            return buf;
            ***************
            *** 1536,1542 ****
            char_u *ffname, *sfname;
            int message;
            {
            ! BUF *buf;

            if (ffname == NULL || *ffname == NUL)
            {
            --- 1569,1578 ----
            char_u *ffname, *sfname;
            int message;
            {
            ! BUF *buf;
            ! #ifdef UNIX
            ! struct stat st;
            ! #endif

            if (ffname == NULL || *ffname == NUL)
            {
            ***************
            *** 1544,1549 ****
            --- 1580,1588 ----
            vim_free(curbuf->b_sfname);
            curbuf->b_ffname = NULL;
            curbuf->b_sfname = NULL;
            + #ifdef UNIX
            + st.st_dev = (unsigned)-1;
            + #endif
            }
            else
            {
            ***************
            *** 1562,1568 ****
            --- 1601,1613 ----
            * - if the buffer is loaded, fail
            * - if the buffer is not loaded, delete it from the list
            */
            + #ifdef UNIX
            + if (mch_stat((char *)ffname, &st) < 0)
            + st.st_dev = (unsigned)-1;
            + buf = buflist_findname_stat(ffname, &st);
            + #else
            buf = buflist_findname(ffname);
            + #endif
            if (buf != NULL && buf != curbuf)
            {
            if (buf->b_ml.ml_mfp != NULL) /* it's loaded, fail */
            ***************
            *** 1588,1594 ****
            }
            curbuf->b_fname = curbuf->b_sfname;
            #ifdef UNIX
            ! buf_setino(curbuf);
            #endif

            #ifndef SHORT_FNAME
            --- 1633,1645 ----
            }
            curbuf->b_fname = curbuf->b_sfname;
            #ifdef UNIX
            ! if (st.st_dev == (unsigned)-1)
            ! curbuf->b_dev = -1;
            ! else
            ! {
            ! curbuf->b_dev = st.st_dev;
            ! curbuf->b_ino = st.st_ino;
            ! }
            #endif

            #ifndef SHORT_FNAME

            --
            OLD WOMAN: King of the WHO?
            ARTHUR: The Britons.
            OLD WOMAN: Who are the Britons?
            "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

            /-/-- Bram Moolenaar --- Bram@... --- http://www.moolenaar.net --\-\
            \-\-- Vim: http://www.vim.org ---- ICCF Holland: http://www.vim.org/iccf --/-/
          • Paul_A
            ... Thanks for that explanation, I hadn t considered the possibility of the file being deleted in the interim. ... I haven t tried any profiling yet, but have
            Message 5 of 6 , Jun 6, 2000
            • 0 Attachment
              On Tue, Jun 06, 2000 at 12:32:06PM +0200, Bram Moolenaar wrote:
              > It looks like the time is spend in the stat() call, which is the call required
              > to get the inode. Inodes are already used to compare files. Vim also stores
              > the absolute path name, because that is what really identifies a file for the
              > user (a file could be deleted and created again, giving it a new inode number,
              > but it's still the same file from the users point of view). This mostly
              > matters for the viminfo feature.

              Thanks for that explanation, I hadn't considered the possibility of the file
              being deleted in the interim.

              > Profiling would still be useful, because it often shows surprising things.
              > For example, it might be that only the first stat() call for a file is slow,
              > and following ones are fast (because of caching). Reducing the number of
              > stat() calls for a file won't help then. You could also use ktrace/kdump to
              > see how much time is spend in each system call.

              I haven't tried any profiling yet, but have given your patch a whirl.
              It looks like your suspicion about the cache effect was correct, eliminating
              the extra stat of each file doesn't seem to help. Strangely enough, it
              mostly measured slower on several repetitions, but it was hardly scientific
              and the numbers were up and down a lot.

              > You could try out the patch below. It halves the number of stat() calls for
              > each file added to the buffer list. Let me know if this significantly reduces
              > the time to start Vim. For me it's a matter of a few percent (starting with
              > about 1500 files takes about three seconds). It could matter a lot more when
              > using a network or clearcase.

              Thanks Bram, I'll let you know if profiling uncovers anything interesting.

              --
              Paul Ackersviller
            Your message has been successfully submitted and would be delivered to recipients shortly.