Loading ...
Sorry, an error occurred while loading the content.

Re: Race condition during file saving

Expand Messages
  • Nikolai Weibull
    ... I haven t really understood what the problem is (I don t believe that there actually is one), but would opening the file exclusively solve the problem?
    Message 1 of 19 , Jan 4, 2009
    • 0 Attachment
      On Sun, Jan 4, 2009 at 15:57, Adam Osuchowski <adwol@...> wrote:

      > open("testfile", O_WRONLY|O_CREAT|O_TRUNC, 0600) = 3

      I haven't really understood what the problem is (I don't believe that
      there actually is one), but would opening the file exclusively solve
      the problem?

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_dev" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • Adam Osuchowski
      ... There is a non-zero time period between open file and write complete content. Because vim truncate file while opening (O_TRUNC), another process which
      Message 2 of 19 , Jan 4, 2009
      • 0 Attachment
        Nikolai Weibull wrote:
        > I haven't really understood what the problem is (I don't believe that
        > there actually is one),

        There is a non-zero time period between open file and write complete
        content. Because vim truncate file while opening (O_TRUNC), another
        process which would like to read it can hit in the moment when file will
        be empty.

        > but would opening the file exclusively solve the problem?

        Do you mean O_EXCL flag to open syscall? It doesn't protect against
        such a situation. It only guarantee that file doesn't exist prior to
        call open in atomic manner.

        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_dev" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • Nikolai Weibull
        ... Yeah, so? That s not a bug. File systems are, generally, not databases with ACID properties. They don t provide transactions (at the level we re
        Message 3 of 19 , Jan 4, 2009
        • 0 Attachment
          On Sun, Jan 4, 2009 at 16:26, Adam Osuchowski <adwol@...> wrote:

          > Nikolai Weibull wrote:

          >> I haven't really understood what the problem is (I don't believe that
          >> there actually is one),

          > There is a non-zero time period between open file and write complete
          > content. Because vim truncate file while opening (O_TRUNC), another
          > process which would like to read it can hit in the moment when file will
          > be empty.

          Yeah, so? That's not a bug. File systems are, generally, not
          databases with ACID properties. They don't provide transactions (at
          the level we're discussing).

          >> but would opening the file exclusively solve the problem?

          > Do you mean O_EXCL flag to open syscall? It doesn't protect against
          > such a situation. It only guarantee that file doesn't exist prior to
          > call open in atomic manner.

          No, I mean both O_EXCL (so that a file hasn't been created in between
          the time the original file has been renamed and the new one opened - a
          case so far not mentioned (or?)) and flock:ing it (but, as I
          understand from the man page, this doesn't seem to be very interesting
          either).

          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_dev" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • Charles E. Campbell, Jr.
          ... If you re talking Unix/Linux/etc -- flock is a cooperative file locking mechanism, and processes which ignore it are not impeded. Look into mandatory file
          Message 4 of 19 , Jan 4, 2009
          • 0 Attachment
            Nikolai Weibull wrote:
            > On Sun, Jan 4, 2009 at 16:26, Adam Osuchowski <adwol@...> wrote:
            >
            >
            >> Nikolai Weibull wrote:
            >>
            >
            >
            >>> I haven't really understood what the problem is (I don't believe that
            >>> there actually is one),
            >>>
            >
            >
            >> There is a non-zero time period between open file and write complete
            >> content. Because vim truncate file while opening (O_TRUNC), another
            >> process which would like to read it can hit in the moment when file will
            >> be empty.
            >>
            >
            > Yeah, so? That's not a bug. File systems are, generally, not
            > databases with ACID properties. They don't provide transactions (at
            > the level we're discussing).
            >
            >
            >>> but would opening the file exclusively solve the problem?
            >>>
            >
            >
            >> Do you mean O_EXCL flag to open syscall? It doesn't protect against
            >> such a situation. It only guarantee that file doesn't exist prior to
            >> call open in atomic manner.
            >>
            >
            > No, I mean both O_EXCL (so that a file hasn't been created in between
            > the time the original file has been renamed and the new one opened - a
            > case so far not mentioned (or?)) and flock:ing it (but, as I
            > understand from the man page, this doesn't seem to be very interesting
            > either).
            >
            If you're talking Unix/Linux/etc -- flock is a cooperative file locking
            mechanism, and processes which ignore it are not impeded.
            Look into mandatory file locking (the mand option to mount). Mandatory
            file locking is kernel enforced, and not even root apparently
            can force its way through.

            Mandatory file locking by vim would not be practical, as it generally
            requires root access to set up, and only involves files on the
            mandatory-filelocking mounted device.

            Regards,
            Chip Campbell


            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_dev" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • Nikolai Weibull
            On Sun, Jan 4, 2009 at 23:19, Charles E. Campbell, Jr. ... Just to make sure, did you read the part where I wrote [flock] doesn t seem to be very interesting
            Message 5 of 19 , Jan 4, 2009
            • 0 Attachment
              On Sun, Jan 4, 2009 at 23:19, Charles E. Campbell, Jr.
              <drchip@...> wrote:

              > Nikolai Weibull wrote:

              >> No, I mean both O_EXCL (so that a file hasn't been created in between
              >> the time the original file has been renamed and the new one opened - a
              >> case so far not mentioned (or?)) and flock:ing it (but, as I
              >> understand from the man page, this doesn't seem to be very interesting
              >> either).

              > If you're talking Unix/Linux/etc -- flock is a cooperative file locking
              > mechanism, and processes which ignore it are not impeded.

              Just to make sure, did you read the part where I wrote "[flock]
              doesn't seem to be very interesting either"?

              I just want to know what your intentions were.

              Was it simply to fill in extra information about why I wrote that?

              Or did it seem like I didn't know what I was talking about and you
              wrote this reply to correct me?

              I'm finding that I get a lot of replys, especially at work, where I
              get the feeling that my mail hasn't really been read through. A
              classic is where you ask someone two questions and you get a reply
              with an answer for the first one. Or where you answer two questions
              and they restate the second one in their reply.

              Either way, I really don't think we have a problem to fix. I /want/
              to be able to read what a program is writing to a file, mid-write.
              Generally, when a long process is executing I want to be able to tell
              what's being written. If you're writing to a file that another
              program critically needs /that's/ your problem. Not whether you made
              sure to set up some intricate locking mechanism or not.

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_dev" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • Charles E. Campbell, Jr.
              ... Nikolai -- you re getting a bit sensitive here -- it was simply to fill in extra information... , as you hadn t explained why flock wasn t interesting.
              Message 6 of 19 , Jan 4, 2009
              • 0 Attachment
                Nikolai Weibull wrote:
                > On Sun, Jan 4, 2009 at 23:19, Charles E. Campbell, Jr.
                > <drchip@...> wrote:
                >
                >
                >> Nikolai Weibull wrote:
                >>
                >
                >
                >>> No, I mean both O_EXCL (so that a file hasn't been created in between
                >>> the time the original file has been renamed and the new one opened - a
                >>> case so far not mentioned (or?)) and flock:ing it (but, as I
                >>> understand from the man page, this doesn't seem to be very interesting
                >>> either).
                >>>
                >
                >
                >> If you're talking Unix/Linux/etc -- flock is a cooperative file locking
                >> mechanism, and processes which ignore it are not impeded.
                >>
                >
                > Just to make sure, did you read the part where I wrote "[flock]
                > doesn't seem to be very interesting either"?
                >
                > I just want to know what your intentions were.
                >
                > Was it simply to fill in extra information about why I wrote that?
                >
                > Or did it seem like I didn't know what I was talking about and you
                > wrote this reply to correct me?
                >
                > I'm finding that I get a lot of replys, especially at work, where I
                > get the feeling that my mail hasn't really been read through. A
                > classic is where you ask someone two questions and you get a reply
                > with an answer for the first one. Or where you answer two questions
                > and they restate the second one in their reply.
                >
                > Either way, I really don't think we have a problem to fix. I /want/
                > to be able to read what a program is writing to a file, mid-write.
                > Generally, when a long process is executing I want to be able to tell
                > what's being written. If you're writing to a file that another
                > program critically needs /that's/ your problem. Not whether you made
                > sure to set up some intricate locking mechanism or not.
                >
                Nikolai -- you're getting a bit sensitive here -- it was "simply to fill
                in extra information...", as you hadn't explained why flock "wasn't
                interesting."
                Plus I added the mandatory kernel-enforced locking information.

                Likely areas for problems like this concern cooperative editing (ie.
                multiple people editing the same file) and editing log files (or other
                files which are potentially being written to by some other program).
                Vim isn't designed for cooperative editing; I seem to recall it being on
                a wishlist, though. Editing log files is problematic because they
                generally aren't using mandatory file locking.

                Personally, I agree with you that there's not a problem to fix; at most,
                there's new behavior (that wishlist stuff) that someone may want. To
                avoid the need for cooperative editing, use cvs/git/etc and use separate
                copies and repositories. Editing log files is likely to remain
                problematic and requires more than just vim to change (unless the
                logging process already uses flock, perhaps I'll look into it sometime).

                Regards,
                Chip Campbell


                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_dev" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • Adam Osuchowski
                ... So what is your advice? Ignore it? It s very comfortable to call flows features, we have perfect situation and don t need to worry about consequences. ...
                Message 7 of 19 , Jan 4, 2009
                • 0 Attachment
                  Nikolai Weibull wrote:
                  > Either way, I really don't think we have a problem to fix.

                  So what is your advice? Ignore it? It's very comfortable to call flows
                  features, we have perfect situation and don't need to worry about
                  consequences.

                  > If you're writing to a file that another program critically needs
                  > /that's/ your problem.

                  Configuration file, for example, is critically for almost every daemon.
                  Do you think that it is only my problem? No, it is very real scenario
                  which could happen to everyone who use vim.

                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_dev" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                • Adam Osuchowski
                  ... I didn t tell about multiple people editing the same file or editing log files. I told about _ONE_ person editing file which could be read by another
                  Message 8 of 19 , Jan 4, 2009
                  • 0 Attachment
                    Charles E. Campbell, Jr. wrote:
                    > Likely areas for problems like this concern cooperative editing (ie.
                    > multiple people editing the same file) and editing log files (or other
                    > files which are potentially being written to by some other program).
                    > Vim isn't designed for cooperative editing; I seem to recall it being on
                    > a wishlist, though. Editing log files is problematic because they
                    > generally aren't using mandatory file locking.

                    I didn't tell about multiple people editing the same file or editing log
                    files. I told about _ONE_ person editing file which could be read by another
                    process at the same time. Isn't vim designed for it too? Don't think so.

                    > To avoid the need for cooperative editing, use cvs/git/etc and use separate
                    > copies and repositories.

                    And keep all /etc files in cvs repo? It's only pinning the blame on another
                    application (cvs, in this case).

                    --~--~---------~--~----~------------~-------~--~----~
                    You received this message from the "vim_dev" maillist.
                    For more information, visit http://www.vim.org/maillist.php
                    -~----------~----~----~----~------~----~------~--~---
                  • Nikolai Weibull
                    ... ? ... That s what crontab -e is for, among other things. --~--~---------~--~----~------------~-------~--~----~ You received this message from the vim_dev
                    Message 9 of 19 , Jan 4, 2009
                    • 0 Attachment
                      On Mon, Jan 5, 2009 at 00:06, Adam Osuchowski <adwol@...> wrote:

                      > Nikolai Weibull wrote:

                      >> Either way, I really don't think we have a problem to fix.

                      > So what is your advice? Ignore it? It's very comfortable to call flows
                      > features, we have perfect situation and don't need to worry about
                      > consequences.

                      ?

                      >> If you're writing to a file that another program critically needs
                      >> /that's/ your problem.

                      > Configuration file, for example, is critically for almost every daemon.
                      > Do you think that it is only my problem? No, it is very real scenario
                      > which could happen to everyone who use vim.

                      That's what crontab -e is for, among other things.

                      --~--~---------~--~----~------------~-------~--~----~
                      You received this message from the "vim_dev" maillist.
                      For more information, visit http://www.vim.org/maillist.php
                      -~----------~----~----~----~------~----~------~--~---
                    • Ben Schmidt
                      ... He didn t mean it s your problem personally and that no other user would experience it. He meant it s your problem as the user, not the editor s problem.
                      Message 10 of 19 , Jan 4, 2009
                      • 0 Attachment
                        >> If you're writing to a file that another program critically needs
                        >> /that's/ your problem.
                        >
                        > Configuration file, for example, is critically for almost every daemon.
                        > Do you think that it is only my problem? No, it is very real scenario
                        > which could happen to everyone who use vim.

                        He didn't mean it's your problem personally and that no other user would
                        experience it. He meant it's your problem as the user, not the editor's
                        problem.

                        If you edit your config files, most likely you need to SIGHUP the daemon
                        to reread it anyway. And if you don't, most likely the daemon is aware
                        of the somewhat dumb practice of reading its config files at
                        unpredictable intervals, and provides a device, such as crontab -e, to
                        avoid problems, as has already been mentioned. Playing games moving
                        temporary files around for the purpose of keeping daemons happy is not
                        an editor's job, but the user's, or a utility such as crontab's.

                        Having the editor write to a temporary file and move it into place has
                        other worse effects as have already been mentioned (breaking of links,
                        use of file descriptors by calling processes such as crontab -e, etc.).
                        Indeed, other software sometimes expects the editor to overwrite the
                        original file, which implies a time with an empty or partially written
                        file.

                        So although the current behaviour isn't really ideal, neither are the
                        alternatives, and in fact, the alternatives are probably worse.

                        It could be another option, I suppose, or incorporated into an existing
                        one (it is only possible if writebackup and backup are unset or
                        copybackup is yes, or there is a time of nonexistence of the file, which
                        is just as bad), but is it worth the bother? How often are people
                        actually bitten by this? Very rarely, I think, and the consequences are
                        far from catastrophic when it happens. It couldn't be the default due to
                        the other negative side effects of the method, too, so would have to be
                        set when appropriate, which means you might as well use some other tool
                        to do the job.

                        Ben.




                        --~--~---------~--~----~------------~-------~--~----~
                        You received this message from the "vim_dev" maillist.
                        For more information, visit http://www.vim.org/maillist.php
                        -~----------~----~----~----~------~----~------~--~---
                      Your message has been successfully submitted and would be delivered to recipients shortly.