Loading ...
Sorry, an error occurred while loading the content.

Re: VIM and NTFS streams

Expand Messages
  • krischik
    ... You misinterpreted the output. Lets try dir /A /F /: C: Documents and Settings ... .vimrc C: Documents and Settings ... .vimrc:
    Message 1 of 25 , Feb 1, 2008
    • 0 Attachment
      On 1 Feb., 09:15, "George V. Reilly" <geo...@...> wrote:
      > On 31/01/2008, krischik <krisc...@...> wrote:

      > > You might want to get yourself the demo version of 4NT (http://
      > >www.jpsoft.com/) and then use "DIR /:" a bit - you might be surprised
      > > how many streams are already used on your system. Most notably:
      >
      > > 12.12.2007 14:03 1'670 _H_A_________ .vimrc
      > > 0
      > > {4c8cc155-6c1e-11d1-8e41-00c04fb9386d}:$DATA
      >
      > Every NTFS file is implemented as one or more streams, and the default
      > stream is called $DATA. It's a bit like finding a directory named . in every
      > directory.

      You misinterpreted the output. Lets try "dir /A /F /:"

      C:\Documents and Settings\...\.vimrc
      C:\Documents and Settings\...\.vimrc:
      {4c8cc155-6c1e-11d1-8e41-00c04fb9386d}

      The stream is called "{4c8cc155-6c1e-11d1-8e41-00c04fb9386d}". But
      speaking of the directory "." - How about:

      >dir /:

      Datenträger in Laufwerk C ist SYSTEM Seriennummer ist
      5cb9:81be
      Verzeichnis von C:\Media\*

      11.10.2007 12:28 <DIR> .
      11.10.2007 12:28 <DIR> ..
      646 $MountMgrRemoteDatabase:$DATA
      28.09.2007 14:43 <JUNCTION> MEDIASAVE
      [Volume{543aeb10-6dbe-11dc-b33a-001a73660515}\]
      11.10.2007 12:28 <JUNCTION> TITANIUM [Volume{95a63aac-7702-11dc-
      b340-001a73660515}\]
      646 Bytes in 0 Dateien und 4 Verzeichnisse
      28'062'420'992 Bytes frei

      or

      >dir /: /F
      C:\Media\.
      C:\Media\..
      C:\Media\..:$MountMgrRemoteDatabase
      C:\Media\MEDIASAVE
      C:\Media\TITANIUM

      Yep - a tream on directory ".." ;-).

      Of course for all the "DIR" commands mentioned you need a command
      interpreter which actually supports streams.

      Martin
      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_dev" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • Alexei Alexandrov
      ... I m not aware well of Linux mechanisms for storing extended attributes inside files on Linux. Word Linux looks strange to me here in fact - it might a
      Message 2 of 25 , Feb 2, 2008
      • 0 Attachment
        krischik wrote:
        > On 31 Jan., 19:57, Alexei Alexandrov <alexei.alexand...@...>
        > wrote:
        >
        >> 2 notes here:
        >
        >> 1. I think NTFS streams is useless feature.
        >
        > NTFS Streams can be used for the very same stuff resource streams on
        > MacOS or extended attributes on OS/2 / Linux are used.

        I'm not aware well of Linux mechanisms for storing extended attributes
        inside files on Linux. Word "Linux" looks strange to me here in fact -
        it might a property of extX filesystem, but it doesn't have to do
        anything with Linux I think since I may use other systems for root mount
        point - reiserfs, for example. Do you have exact answers to the
        following questions:
        * Which mechanism is used on Linux to store these extended attributes on
        Linux? Is it filesystem-based? If yes, which filesystems currently have
        it implemented?
        * Which applications use this feature on Linux? Are they OS-specific?
        Are there application programs that use this? Are those programs
        portable? If yes, how they deal with absence of this feature on other
        file systems?

        These are exactly questions which I would ask myself if I would be
        designing something new and would be considering using NTFS file
        streams. Will there be FAT32 file system clients? Would I care to port
        the program to other system some time later? Et cetera. And most (if not
        all) programs choose to use simple abstractions (files/directories)
        available on all modern systems because it works well, because you'd
        better keep it simple and because there are more important things to
        focus on.

        >
        >> I've never seen any
        >> practical example of the usefulness of this.
        >
        > You might want to get yourself the demo version of 4NT (http://
        > www.jpsoft.com/) and then use "DIR /:" a bit - you might be surprised
        > how many streams are already used on your system. Most notably:
        >
        > 12.12.2007 14:03 1'670 _H_A_________ .vimrc
        > 0
        > {4c8cc155-6c1e-11d1-8e41-00c04fb9386d}:$DATA
        >

        This looks like a default data stream in this file. I don't see that
        there are 2 streams here - am I overlooking something?

        P.S. I do see some cases where the data streams feature might be useful.
        For example, anti-virus program might store some information about the
        scanned file in a separate stream. But when I think of it more I realize
        that even in these cases data streams approach would be questionable and
        that there other ways to implement it with potentially better
        performance and less limitations. For example, you cannot attach a new
        data stream to a file which is available to you as read-only.

        P.P.S. Thanks for reading to this point! :) Sorry for a long post and
        somewhat clumsy English.

        --
        Alexei Alexandrov


        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_dev" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • krischik
        On 2 Feb., 19:47, Alexei Alexandrov ... Open a console and type man 5 attr ? Or look here: http://linux.die.net/man/5/attr ...
        Message 3 of 25 , Feb 4, 2008
        • 0 Attachment
          On 2 Feb., 19:47, Alexei Alexandrov <alexei.alexand...@...>
          wrote:
          > krischik wrote:
          > > On 31 Jan., 19:57, Alexei Alexandrov <alexei.alexand...@...>
          > > wrote:
          >
          > >> 2 notes here:
          >
          > >> 1. I think NTFS streams is useless feature.
          >
          > > NTFS Streams can be used for the very same stuff resource streams on
          > > MacOS or extended attributes on OS/2 / Linux are used.
          >
          > I'm not aware well of Linux mechanisms for storing extended attributes
          > inside files on Linux.

          Open a console and type "man 5 attr"? Or look here:

          http://linux.die.net/man/5/attr

          > Word "Linux" looks strange to me here in fact -
          > it might a property of extX filesystem, but it doesn't have to do
          > anything with Linux I think since I may use other systems for root mount
          > point - reiserfs, for example.

          No it has nothing to do with Linux. FreeBSD, Solaris and Mac OS X
          support extended attributes as well. See:

          http://en.wikipedia.org/wiki/Extended_file_attributes

          > Do you have exact answers to the
          > following questions:

          > * Which mechanism is used on Linux to store these extended attributes on
          > Linux? Is it filesystem-based? If yes, which filesystems currently have
          > it implemented?

          Quote Wikipedia: "In Linux, the ext2, ext3, ext4, JFS, ReiserFS and
          XFS filesystems support extended attributes (abbreviated xattr) if the
          libattr feature is enabled in the kernel configuration. "

          Which it is ever since about SuSE 9.0

          > * Which applications use this feature on Linux? Are they OS-specific?

          Sadly I know only of two applications: setxattr and getxattr. But they
          are not OS specific - they are supported via the "libattr" library.


          > Are there application programs that use this? Are those programs
          > portable? If yes, how they deal with absence of this feature on other
          > file systems? These are exactly questions which I would ask myself if I would be
          > designing something new and would be considering using NTFS file
          > streams.

          These are also exactly the questions which hold back software
          engineering for the last 20 years or so.

          > Will there be FAT32 file system clients? Would I care to port
          > the program to other system some time later? Et cetera. And most (if not
          > all) programs choose to use simple abstractions (files/directories)
          > available on all modern systems because it works well, because you'd
          > better keep it simple and because there are more important things to
          > focus on.

          On the other hand use of extended attributes could solve a problem
          with 5 lines of code where solving the same problem without could cost
          you 50. Determine file types, text file line endings and text file
          encoding come to my mind here. Ask Bram how many line of code he
          needed in Vim to determine these three informations. With consequent
          use of xattribs it would have been 6 lines:

          char mime_type[64];
          char mime_encoding[64];
          char line_ending[64];

          getxattr (filename, "Content-Type",mime_type, sizeof mime_type);
          getxattr (filename, "Line-Ending", line_ending, sizeof line_ending);
          getxattr (filename, "Content-Encoding",mime_encoding, sizeof
          mime_encoding);

          And best of all: you know before you open the file. AFAIK Bram need to
          close and reopen files in unfortunate combinations.

          It is often overlooked that not only using a new feature posses risks
          but not using as well.

          How often I ended up with xxx.yyy.htm after a download... Or ever
          better: the amount of frustrated SymbianOS users which tried to
          download a theme (named xxx.utz) and ended up with xxx.zip. All
          because some webserver and or browser made a mistake when trying to
          determining the file type (utz and zip share the same file header).

          And it is precisely those "simple abstractions (files/directories)
          available on all modern systems" which bring those problems. Only:
          It's not "modern systems" - it's "between 1 and 2 decades old
          systems". Truly modern operating systems support extended attributes.

          > P.S. I do see some cases where the data streams feature might be useful.
          > For example, anti-virus program might store some information about the
          > scanned file in a separate stream.

          To late: there are already anti virus solutions which do that.

          Martin
          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_dev" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • krischik
          ... One of the reasons I feel so strongly about this issue is the fact that I used OS/2 before where storing the file type in the extended attributes was
          Message 4 of 25 , Feb 4, 2008
          • 0 Attachment
            On 4 Feb., 15:18, krischik <krisc...@...> wrote:

            > And it is precisely those "simple abstractions (files/directories)
            > available on all modern systems"  which bring those problems. Only:
            > It's not "modern systems" - it's "between 1 and 2 decades old
            > systems". Truly modern operating systems support extended attributes.

            One of the reasons I feel so strongly about this issue is the fact
            that I used OS/2 before where storing the file type in the extended
            attributes was standart. And it worked a lot better then extensions
            (Dos) or scanning the first 256 bytes for magic patterns (Unix). There
            was no guessing involved: The first filetype (the .TYPE xattrib in OS/
            2 is a list) was the primary filetype the rest where possible
            alternative representations (like a html file could also be
            represented as plain text).

            And then there was the .ICON xattrib - how I miss the .ICON xattrib -
            I could drag and drop an icon to every file and from thereon the file
            would be represented by that icon.

            Now, nostalgia besides: The point is that those "simple abstractions"
            lead to primitive solutions. And indeed OS/2 with it's advanced file
            system offered a user experience which even Vista won't offer you. KDE
            offers some of it - for the price of littering you filesystem with
            ".directory" files. But KDE still can't do what OS/2 could do - mainly
            because they would need use extended attributes to do so - and the KDE
            developers follow the same line of thought you do - which in turn
            holds them back.

            Martin

            PS: On FAT the extended attributes where stored in a file called "EA
            DATA. SF" - which shows that even older file systems can be taught new
            tricks.
            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_dev" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • Matt Wozniski
            ... While this would be nice, it would require support code from every application you have. It may only be 6 lines, but 6 lines * 5000 binaries is much more
            Message 5 of 25 , Feb 4, 2008
            • 0 Attachment
              On Feb 4, 2008 9:18 AM, krischik wrote:
              >
              > On the other hand use of extended attributes could solve a problem
              > with 5 lines of code where solving the same problem without could cost
              > you 50. Determine file types, text file line endings and text file
              > encoding come to my mind here. Ask Bram how many line of code he
              > needed in Vim to determine these three informations. With consequent
              > use of xattribs it would have been 6 lines:
              >
              > char mime_type[64];
              > char mime_encoding[64];
              > char line_ending[64];
              >
              > getxattr (filename, "Content-Type",mime_type, sizeof mime_type);
              > getxattr (filename, "Line-Ending", line_ending, sizeof line_ending);
              > getxattr (filename, "Content-Encoding",mime_encoding, sizeof
              > mime_encoding);

              While this would be nice, it would require support code from every
              application you have. It may only be 6 lines, but 6 lines * 5000
              binaries is much more code than is in vim for line ending detection.
              After all, vim can't use this information unless something put it
              there, which requires that everything that can create a file - from
              touch to wget to shell redirection - needs to be able to put that
              attribute into the file. And, AFAIK, there's no way for those wget or
              shell redirection to even know what type of line ending the data that
              they wrote out had. Which means they'd need to detect it. Which
              would require that just about every program out there that is
              transcribing a data stream from one spot to another, rather than
              authoring it itself, would need to duplicate the sort of EOL detection
              code in vim in itself. And vim would still need to continue
              supporting the old way since it's designed to run on filesystems that
              still don't support extended attributes - or it would need to come up
              with a way to fake extended attributes on those FS types. No one is
              saying that extended attributes can't do useful things - just that
              without being in any way standardized, it's unreasonable to expect
              that any attributes will ever be set by anyone but you.

              That being said, if vim had a convenient way to see if extended
              attributes support was available on the filesystem, it might be worth
              caching things like the fileformat in the extended attributes, so it
              only needed to be computed once... Though that might open up a whole
              new can of worms, since someone could easily change the line endings
              with another tool between two vim runs, confusing the n00b user with a
              bunch of ^M's show up everywhere... all in all, this is one place
              where manual detection seems the best idea.

              For things like content encoding, it might be more useful - but again,
              since every other tool doesn't create it, vim would still need to fall
              back on autodetection if it couldn't find an attribute for it.

              ~Matt

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_dev" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • krischik
              ... But the other 5000 applications need detection as well. I did mention webserver and browser getting detection wrong. konquror/nautilus/ explorer needs
              Message 6 of 25 , Feb 4, 2008
              • 0 Attachment
                On 4 Feb., 21:10, "Matt Wozniski" <m...@...> wrote:
                > On Feb 4, 2008 9:18 AM, krischik wrote:
                >
                >
                >
                > > On the other hand use of extended attributes could solve a problem
                > > with 5 lines of code where solving the same problem without could cost
                > > you 50. Determine file types, text file line endings and text file
                > > encoding come to my mind here. Ask Bram how many line of code he
                > > needed in Vim to determine these three informations. With consequent
                > > use of xattribs it would have been 6 lines:
                >
                > > char mime_type[64];
                > > char mime_encoding[64];
                > > char line_ending[64];
                >
                > > getxattr (filename, "Content-Type",mime_type, sizeof mime_type);
                > > getxattr (filename, "Line-Ending", line_ending, sizeof line_ending);
                > > getxattr (filename, "Content-Encoding",mime_encoding, sizeof
                > > mime_encoding);
                >
                > While this would be nice, it would require support code from every
                > application you have.  It may only be 6 lines, but 6 lines * 5000
                > binaries is much more code than is in vim for line ending detection.

                But the other 5000 applications need detection as well. I did mention
                webserver and browser getting detection wrong. konquror/nautilus/
                explorer needs detection and far more complex then vim. And sure OS/2
                needed detection. But there you could always overwrite faulty detected
                value by manualy correcting the EA's.

                > After all, vim can't use this information unless something put it
                > there, which requires that everything that can create a file - from
                > touch to wget to shell redirection - needs to be able to put that
                > attribute into the file.  And, AFAIK, there's no way for those wget or
                > shell redirection to even know what type of line ending the data that
                > they wrote out had.  Which means they'd need to detect it.

                Just one example: The OS/2 version of ZIP would pack extended
                attributes. Once it catches on then more application will support it.
                But I know that this won't happen. We went down the "worse is better"
                way for far to long.

                > Which
                > would require that just about every program out there that is
                > transcribing a data stream from one spot to another, rather than
                > authoring it itself, would need to duplicate the sort of EOL detection
                > code in vim in itself.  And vim would still need to continue
                > supporting the old way since it's designed to run on filesystems that
                > still don't support extended attributes - or it would need to come up
                > with a way to fake extended attributes on those FS types.  No one is
                > saying that extended attributes can't do useful things - just that
                > without being in any way standardized, it's unreasonable to expect
                > that any attributes will ever be set by anyone but you.

                Ahh, indeed, there is the problem and the reason why computing has not
                advanced as much in the last 15 years as one would have expected
                seeing the 15 years before: Backward compatibilty has hobbled us.

                Martin
                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_dev" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • Matt Wozniski
                ... I m not sure that most other apps do need detection. wget, for instance, doesn t have to care what line endings the data it saves has. But, for what you
                Message 7 of 25 , Feb 4, 2008
                • 0 Attachment
                  On Feb 4, 2008 4:29 PM, krischik wrote:
                  >
                  > On 4 Feb., 21:10, "Matt Wozniski" wrote:
                  > >
                  > > While this would be nice, it would require support code from every
                  > > application you have. It may only be 6 lines, but 6 lines * 5000
                  > > binaries is much more code than is in vim for line ending detection.
                  >
                  > But the other 5000 applications need detection as well. I did mention
                  > webserver and browser getting detection wrong. konquror/nautilus/
                  > explorer needs detection and far more complex then vim. And sure OS/2
                  > needed detection. But there you could always overwrite faulty detected
                  > value by manualy correcting the EA's.

                  I'm not sure that most other apps do need detection. wget, for
                  instance, doesn't have to care what line endings the data it saves
                  has. But, for what you want, it would have to detect and save it,
                  even though it doesn't use it. Shell redirection has no idea what's
                  going through it, so it has no way to possibly say what Content-type
                  it is... or dd... or tar extracting files that were created on a
                  filesystem that didn't track extended attributes...

                  > > After all, vim can't use this information unless something put it
                  > > there, which requires that everything that can create a file - from
                  > > touch to wget to shell redirection - needs to be able to put that
                  > > attribute into the file. And, AFAIK, there's no way for those wget or
                  > > shell redirection to even know what type of line ending the data that
                  > > they wrote out had. Which means they'd need to detect it.
                  >
                  > Just one example: The OS/2 version of ZIP would pack extended
                  > attributes. Once it catches on then more application will support it.
                  > But I know that this won't happen. We went down the "worse is better"
                  > way for far to long.

                  The issue isn't waiting for it to catch on, it's that until it's
                  universally available it only increases the amount of code and
                  maintenance burden. And, there's no way that it would ever be
                  universally available for things like line endings style, because most
                  applications just don't know or need to care, so have no reason to set
                  that attribute. Things like marking the content encoding might be
                  more useful, but still, not good to work with... If, for example, you
                  had a file "test.txt" encoded in UTF-16 and with extended attributes
                  marking it as such, and your locale is set to use UTF-8, what would
                  you expect the result of "cp test.txt test2.txt" to be? What about
                  "cat text.txt >text2.txt"? If you expect those two commands to have
                  the same effect, I don't see how it can be done without changes to cp
                  (mark attrs on dest), cat (mark attrs on stdout), and the kernel
                  itself (allow extended attributes on streams).

                  > > Which
                  > > would require that just about every program out there that is
                  > > transcribing a data stream from one spot to another, rather than
                  > > authoring it itself, would need to duplicate the sort of EOL detection
                  > > code in vim in itself. And vim would still need to continue
                  > > supporting the old way since it's designed to run on filesystems that
                  > > still don't support extended attributes - or it would need to come up
                  > > with a way to fake extended attributes on those FS types. No one is
                  > > saying that extended attributes can't do useful things - just that
                  > > without being in any way standardized, it's unreasonable to expect
                  > > that any attributes will ever be set by anyone but you.
                  >
                  > Ahh, indeed, there is the problem and the reason why computing has not
                  > advanced as much in the last 15 years as one would have expected
                  > seeing the 15 years before: Backward compatibilty has hobbled us.

                  The problem with this idea isn't the limitations inherent in backwards
                  compatibility - if it were worthwhile to do, it's no problem to have
                  code that uses attributes if possible and the old, backwards
                  compatible method if not. The problem is that creating this standard
                  and tying it properly into a POSIX OS would require a lot of low-level
                  changes. If we can agree on the requirements - that there must be
                  some number of predefined attributes that are set on every regular
                  file (and what about special files?), that the operating system must
                  handle initializing these attributes correctly when a file is created,
                  that streams would also need to have attributes, and that copying,
                  extracting, and in general "moving around" data will "just work", can
                  you see how difficult this would be to implement? To fully implement
                  this, the very least that's required is an operating system that only
                  supports filesystems that have some way of storing extended
                  attributes, a kernel that's able to initialize each of those
                  attributes on every file created, a shell that knows how to get/set
                  these attributes (and which it needs to set), and a copy of the
                  coreutils that are aware of the changes (so that "cp file1 file2"
                  creates file2 with the attrs of file1, not the defaults), at the very
                  least? And that work is just the baseline - the point where other
                  software can start counting on a base set of attributes that will
                  always exist, and trusting that they're correct. On a quick glance, I
                  see absolutely no way to do this properly without support from the OS.

                  ~Matt

                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_dev" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                • Ben Schmidt
                  ... Indeed, I think it has. The Mac OS used to use resource forks and type attributes which were beautiful. Now we have descended to the level of extensions
                  Message 8 of 25 , Feb 4, 2008
                  • 0 Attachment
                    > Ahh, indeed, there is the problem and the reason why computing has not
                    > advanced as much in the last 15 years as one would have expected
                    > seeing the 15 years before: Backward compatibilty has hobbled us.

                    Indeed, I think it has. The Mac OS used to use resource forks and type attributes
                    which were beautiful. Now we have descended to the level of extensions and magic
                    numbers like everyone else. The filesystem supports resource forks and named
                    forks, but nobody's really using them yet--indeed, if anything, at the moment they
                    add confusion because some parts of the OS detect file types that way rather than
                    by extension, when available, so it's hard to know what's going on.

                    Ben.




                    Send instant messages to your online friends http://au.messenger.yahoo.com


                    --~--~---------~--~----~------------~-------~--~----~
                    You received this message from the "vim_dev" maillist.
                    For more information, visit http://www.vim.org/maillist.php
                    -~----------~----~----~----~------~----~------~--~---
                  • Ben Schmidt
                    ... And that sad story told, I do agree with the other posters who don t think this has much use in Vim. I think to have some kind of consistency, probably an
                    Message 9 of 25 , Feb 4, 2008
                    • 0 Attachment
                      Ben Schmidt wrote:
                      >> Ahh, indeed, there is the problem and the reason why computing has not
                      >> advanced as much in the last 15 years as one would have expected
                      >> seeing the 15 years before: Backward compatibilty has hobbled us.
                      >
                      > Indeed, I think it has. The Mac OS used to use resource forks and type attributes
                      > which were beautiful. Now we have descended to the level of extensions and magic
                      > numbers like everyone else. The filesystem supports resource forks and named
                      > forks, but nobody's really using them yet--indeed, if anything, at the moment they
                      > add confusion because some parts of the OS detect file types that way rather than
                      > by extension, when available, so it's hard to know what's going on.

                      And that sad story told, I do agree with the other posters who don't think this
                      has much use in Vim. I think to have some kind of consistency, probably an OS or
                      other fairly central component needs to lead the way, not a text editor.

                      Ben.




                      Send instant messages to your online friends http://au.messenger.yahoo.com


                      --~--~---------~--~----~------------~-------~--~----~
                      You received this message from the "vim_dev" maillist.
                      For more information, visit http://www.vim.org/maillist.php
                      -~----------~----~----~----~------~----~------~--~---
                    • krischik
                      ... No, a EA aware wget would request the EA s already attached to the file from an EA aware ftp server - the same way it request file permissions today when
                      Message 10 of 25 , Feb 5, 2008
                      • 0 Attachment
                        On 4 Feb., 23:09, "Matt Wozniski" <m...@...> wrote:
                        > On Feb 4, 2008 4:29 PM, krischik wrote:
                        >
                        > > On 4 Feb., 21:10, "Matt Wozniski" wrote:
                        >
                        > > > While this would be nice, it would require support code from every
                        > > > application you have. It may only be 6 lines, but 6 lines * 5000
                        > > > binaries is much more code than is in vim for line ending detection.
                        >
                        > > But the other 5000 applications need detection as well. I did mention
                        > > webserver and browser getting detection wrong. konquror/nautilus/
                        > > explorer needs detection and far more complex then vim. And sure OS/2
                        > > needed detection. But there you could always overwrite faulty detected
                        > > value by manualy correcting the EA's.
                        >
                        > I'm not sure that most other apps do need detection. wget, for
                        > instance, doesn't have to care what line endings the data it saves
                        > has. But, for what you want, it would have to detect and save it,
                        > even though it doesn't use it.

                        No, a EA aware wget would request the EA's already attached to the
                        file from an EA aware ftp server - the same way it request file
                        permissions today when used with "--preserve-permissions".

                        > Shell redirection has no idea what's
                        > going through it, so it has no way to possibly say what Content-type
                        > it is... or dd...

                        That's true.

                        > or tar extracting files that were created on a
                        > filesystem that didn't track extended attributes...

                        Well, you might like to ask the authors - xattr and acl support has
                        been added to tar last year:

                        http://www.redhatmagazine.com/2007/07/02/tips-from-an-rhce-tar-vs-star-the-battle-of-xattrs/

                        > The issue isn't waiting for it to catch on, it's that until it's
                        > universally available it only increases the amount of code and
                        > maintenance burden.

                        You are proving my point: backward compatibility has hobbled software
                        development.

                        > Things like marking the content encoding might be
                        > more useful, but still, not good to work with... If, for example, you
                        > had a file "test.txt" encoded in UTF-16 and with extended attributes
                        > marking it as such, and your locale is set to use UTF-8, what would
                        > you expect the result of "cp test.txt test2.txt" to be?

                        I would use "cp --archive test.txt test2.txt" in which case the EA's
                        are copied as well. At least on SuSE Linux 9.2 onwards. And of course
                        the file would still be UTF-16 - after all it's "copy" not "convert".

                        Without meaning offence: You should read up a little on the subject as
                        you knowledge is not up to date.

                        > What about
                        > "cat text.txt >text2.txt"? If you expect those two commands to have
                        > the same effect, I don't see how it can be done without changes to cp
                        > (mark attrs on dest), cat (mark attrs on stdout), and the kernel
                        > itself (allow extended attributes on streams).

                        I do not expect that cat and cp behave the same. For the simple reason
                        that they never have behaved the same: cp has options like "--
                        preserve" and "--archive" - cat has not. Even today using cat to copy
                        a file will mean that you loose the all the meta informations
                        attached.

                        > a shell that knows how to get/set
                        > these attributes (and which it needs to set),

                        Done: see setfattr and getfattr.

                        > and a copy of the
                        > coreutils that are aware of the changes (so that "cp file1 file2"
                        > creates file2 with the attrs of file1, not the defaults),

                        Done: GNU cp will do that if --archive is used.

                        > On a quick glance, I
                        > see absolutely no way to do this properly without support from the OS.

                        While I countered most of your arguments - you are absolutely right
                        here. Only I have once used an operating system (OS/2) which did
                        exactly that - hence my frustration.

                        Martin
                        --~--~---------~--~----~------------~-------~--~----~
                        You received this message from the "vim_dev" maillist.
                        For more information, visit http://www.vim.org/maillist.php
                        -~----------~----~----~----~------~----~------~--~---
                      • krischik
                        ... I never meant that - I was just countering the claim that named forks are useless. And I only used Vim as an example of where Vim would have been simpler
                        Message 11 of 25 , Feb 5, 2008
                        • 0 Attachment
                          On 5 Feb., 00:12, Ben Schmidt <mail_ben_schm...@...> wrote:

                          > And that sad story told, I do agree with the other posters who don't think this
                          > has much use in Vim. I think to have some kind of consistency, probably an OS or
                          > other fairly central component needs to lead the way, not a text editor.

                          I never meant that - I was just countering the claim that "named
                          forks" are useless. And I only used Vim as an example of where Vim
                          would have been simpler to implement - on an operating system which
                          properly supports "type attributes".

                          Sad that everybody else only sees only hurdles and risks of change and
                          not the chances and the risk of no-change.

                          And yes: there is a rist of no-change!

                          The risk of an ever more complex software world with ever more complex
                          solutions which could be done much easier if one would have done it
                          right from the beginning.

                          And the risk of ever more exploits of those complex solutions.

                          Martin
                          --~--~---------~--~----~------------~-------~--~----~
                          You received this message from the "vim_dev" maillist.
                          For more information, visit http://www.vim.org/maillist.php
                          -~----------~----~----~----~------~----~------~--~---
                        • Tony Mechelynck
                          krischik wrote: [...] ... [...] ... [...] Hmm... According to info cp (on my openSUSE 10.3 system), -a or --archive is equivalent to -dpPR, which means: -d
                          Message 12 of 25 , Feb 5, 2008
                          • 0 Attachment
                            krischik wrote:
                            [...]
                            > I would use "cp --archive test.txt test2.txt" in which case the EA's
                            > are copied as well. At least on SuSE Linux 9.2 onwards. And of course
                            > the file would still be UTF-16 - after all it's "copy" not "convert".
                            >
                            > Without meaning offence: You should read up a little on the subject as
                            > you knowledge is not up to date.
                            [...]
                            > Done: GNU cp will do that if --archive is used.
                            [...]

                            Hmm... According to "info cp" (on my openSUSE 10.3 system), -a or --archive is
                            equivalent to -dpPR, which means:

                            -d copy symlinks as symlinks and preserve hardlinks between sources in the copies
                            -p preserve attributes. If not specifying which attributes, the default is:
                            mode,ownership,timestamps (xattrs must be specified explicitly to be included)
                            -P copy symlinks as symlinks (sic)
                            -R copy directories recursively


                            Best regards,
                            Tony.
                            --
                            "She said, `I know you ... you cannot sing'. I said, `That's nothing,
                            you should hear me play piano.'"
                            -- Morrisey


                            --~--~---------~--~----~------------~-------~--~----~
                            You received this message from the "vim_dev" maillist.
                            For more information, visit http://www.vim.org/maillist.php
                            -~----------~----~----~----~------~----~------~--~---
                          • krischik
                            On 5 Feb., 11:37, Tony Mechelynck ... Well I did not read the manual but copied a file with xattr (Samba creates them at mass)
                            Message 13 of 25 , Feb 5, 2008
                            • 0 Attachment
                              On 5 Feb., 11:37, Tony Mechelynck <antoine.mechely...@...>
                              wrote:
                              > krischik wrote:
                              >
                              > [...]> I would use "cp --archive test.txt test2.txt" in which case the EA's
                              > > are copied as well. At least on SuSE Linux 9.2 onwards. And of course
                              > > the file would still be UTF-16 - after all it's "copy" not "convert".
                              >
                              > > Without meaning offence: You should read up a little on the subject as
                              > > you knowledge is not up to date.
                              > [...]
                              > > Done: GNU cp will do that if --archive is used.
                              >
                              > [...]
                              >
                              > Hmm... According to "info cp" (on my openSUSE 10.3 system), -a or --archive is
                              > equivalent to -dpPR, which means:
                              >
                              > -d copy symlinks as symlinks and preserve hardlinks between sources in the copies
                              > -p preserve attributes. If not specifying which attributes, the default is:
                              > mode,ownership,timestamps (xattrs must be specified explicitly to be included)
                              > -P copy symlinks as symlinks (sic)
                              > -R copy directories recursively

                              Well I did not read the manual but copied a file with xattr (Samba
                              creates them at mass) and saw what happened. But that was on a fairly
                              old SuSE 9.2. I check SuSE 10.3 when I am home.

                              Martin


                              --~--~---------~--~----~------------~-------~--~----~
                              You received this message from the "vim_dev" maillist.
                              For more information, visit http://www.vim.org/maillist.php
                              -~----------~----~----~----~------~----~------~--~---
                            • Paul LeoNerd Evans
                              On Tue, 5 Feb 2008 02:25:26 -0800 (PST) ... I have to agree with krischik here. At University, I once got stuck in a problem with some of the other
                              Message 14 of 25 , Feb 5, 2008
                              • 0 Attachment
                                On Tue, 5 Feb 2008 02:25:26 -0800 (PST)
                                krischik <krischik@...> wrote:

                                > Sad that everybody else only sees only hurdles and risks of change and
                                > not the chances and the risk of no-change.
                                >
                                > And yes: there is a rist of no-change!

                                I have to agree with krischik here. At University, I once got stuck in a
                                problem with some of the other mathematicians, who were all of the
                                opinion that "Yes, I'll have tea if you're having tea". Problem was,
                                nobody actually had tea. It took one of the lawyers to stand up and say
                                "Alright then, I'll have tea." Then all the mathmos jumped ship.

                                An amusing story perhaps, but it proves the point. In all things like
                                this, someone needs to jump first. Does it really matter who that is, as
                                long as someone does?

                                It's been a bone of contention of mine for years, the way modified keys
                                work in terminals. Five years ago XTerm gained ways to express generic
                                modified keys (e.g. Ctrl-Shift-Left). Only recently did any application,
                                such as Vim, start to understand those. It's always been a "you jump,
                                I'll jump" situation - why should the terminal send sequences nobody
                                would understand, or why should any application look for sequences nobody
                                would send? Someone has to go first.

                                Back to the subject of EAs - someone already has gone first. The GNU
                                fileutils already support EAs. As does tar. I also know that the lighttpd
                                webserver uses them by default, only falling back on filename-based
                                detection if the file doesn't have a "Content-Type" EA.

                                It'd be lovely if Vim could set those for it.

                                --
                                Paul "LeoNerd" Evans

                                leonerd@...
                                ICQ# 4135350 | Registered Linux# 179460
                                http://www.leonerd.org.uk/
                              • krischik
                                ... Funny as you say - on my research for my little rants here (yes I do research before I rant) I found out that Apache too uses EA s when to store and detect
                                Message 15 of 25 , Feb 5, 2008
                                • 0 Attachment
                                  On 5 Feb., 12:55, Paul LeoNerd Evans <leon...@...> wrote:

                                  > I also know that the lighttpd
                                  > webserver uses them by default, only falling back on filename-based
                                  > detection if the file doesn't have a "Content-Type" EA.

                                  Funny as you say - on my research for my little rants here (yes I do
                                  research before I rant) I found out that Apache too uses EA's when to
                                  store and detect file types.

                                  And then there is Samba which uses "user.DOSATTRIB" to store dos
                                  attributes.

                                  > It'd be lovely if Vim could set those for it.

                                  That would be a nice indeed. - Bram - are you still with us?

                                  Martin
                                  --~--~---------~--~----~------------~-------~--~----~
                                  You received this message from the "vim_dev" maillist.
                                  For more information, visit http://www.vim.org/maillist.php
                                  -~----------~----~----~----~------~----~------~--~---
                                • Alexei Alexandrov
                                  ... Oh, thanks - this is a good learning for me. It s a shame I never heard of this before. ... Such attributes would be useful, yes. But it s backward
                                  Message 16 of 25 , Feb 5, 2008
                                  • 0 Attachment
                                    krischik wrote:
                                    >
                                    > Open a console and type "man 5 attr"? Or look here:
                                    >
                                    > http://linux.die.net/man/5/attr
                                    >
                                    >
                                    > No it has nothing to do with Linux. FreeBSD, Solaris and Mac OS X
                                    > support extended attributes as well. See:
                                    >
                                    > http://en.wikipedia.org/wiki/Extended_file_attributes
                                    >
                                    > Quote Wikipedia: "In Linux, the ext2, ext3, ext4, JFS, ReiserFS and
                                    > XFS filesystems support extended attributes (abbreviated xattr) if the
                                    > libattr feature is enabled in the kernel configuration. "

                                    Oh, thanks - this is a good learning for me. It's a shame I never heard
                                    of this before.

                                    >
                                    > On the other hand use of extended attributes could solve a problem
                                    > with 5 lines of code where solving the same problem without could cost
                                    > you 50. Determine file types, text file line endings and text file
                                    > encoding come to my mind here. Ask Bram how many line of code he
                                    > needed in Vim to determine these three informations. With consequent
                                    > use of xattribs it would have been 6 lines:
                                    >
                                    > [...]
                                    >
                                    > And best of all: you know before you open the file. AFAIK Bram need to
                                    > close and reopen files in unfortunate combinations.
                                    >

                                    Such attributes would be useful, yes. But it's backward compatibility
                                    that is the root of all evil and you also mentioned this in a sibling
                                    post. So I'm not sure these things will become popular unless there is a
                                    brand new OS appears which doesn't have to be compatible with anything
                                    and it wins the market.

                                    Also, the named attributes story doesn't have to do much with numbered
                                    NTFS data streams, does it? It might be similar but numbers are too far
                                    from string identifier convenience. Which brings me back to my original
                                    thought - the design and implementation of data streams in NTFS are useless.

                                    --
                                    Alexei Alexandrov


                                    --~--~---------~--~----~------------~-------~--~----~
                                    You received this message from the "vim_dev" maillist.
                                    For more information, visit http://www.vim.org/maillist.php
                                    -~----------~----~----~----~------~----~------~--~---
                                  • krischik
                                    On 5 Feb., 22:54, Alexei Alexandrov ... I fear you missed something - since NTFS v3.0 named forks are supported. They are called
                                    Message 17 of 25 , Feb 5, 2008
                                    • 0 Attachment
                                      On 5 Feb., 22:54, Alexei Alexandrov <alexei.alexand...@...>
                                      wrote:

                                      > Also, the named attributes story doesn't have to do much with numbered
                                      > NTFS data streams, does it?

                                      I fear you missed something - since NTFS v3.0 named forks are
                                      supported. They are called "Alternate data streams". See

                                      http://en.wikipedia.org/wiki/NTFS#Features

                                      Martin
                                      --~--~---------~--~----~------------~-------~--~----~
                                      You received this message from the "vim_dev" maillist.
                                      For more information, visit http://www.vim.org/maillist.php
                                      -~----------~----~----~----~------~----~------~--~---
                                    • Bram Moolenaar
                                      ... Vim doesn t detect the situation that you are editing an info stream directly. ... The writebackup option matters too. ... Detecting the colon should be
                                      Message 18 of 25 , Feb 6, 2008
                                      • 0 Attachment
                                        Alex Jakushev wrote:

                                        > On 1/31/08, Bram Moolenaar <Bram@...> wrote:
                                        > >
                                        > > Try setting the 'backupcopy' option to "yes".
                                        >
                                        > This worked, but then it raises another question. Previous value
                                        > of the 'backupcopy' option was auto, which means yes or no, which
                                        > works best. Why didn't it choose yes, if no fails?

                                        Vim doesn't detect the situation that you are editing an info stream
                                        directly.

                                        > Also, in my configuration, backup option is no. Theoretically, I should
                                        > not bother with all this stuff at all? I use Vim7.1 last official release.

                                        The 'writebackup' option matters too.

                                        > > I thought Vim did copy the streams, but I suppose this doesn't work when
                                        > > you edit one specific stream. If you want to look at it: function
                                        > > copy_infostreams() in src/os_win32.c
                                        >
                                        > This may take a while for me to set up everything...
                                        >
                                        > But i have an idea that the problem is with file names.
                                        > When you open some specific stream, vim considers the stream name as
                                        > filename, and filename as subfolder. The stream name is separated from file
                                        > name with a colon (c:\path\foo.txt:bar), and the same colon is used to
                                        > separate drive letter. Maybe this causes some confusion?

                                        Detecting the colon should be simple. When not using NTFS the file name
                                        would be illegal. I'll add a todo item for this. But it would be nice
                                        if someone can make a patch for it.

                                        > Also, VIM converts dots in file name to underscores sometimes, when
                                        > streams are specified.

                                        I haven't seen this. Perhaps it's because the long file name is
                                        converted to a 8.3 file name? Can you give a reproducable example?

                                        > > NTFS streams are mostly restricted to MS-Windows applications, and
                                        > > rarely used. I don't think it's a good idea to support them in Vim
                                        > > directly.
                                        >
                                        > I understand it, but vim has specific windows functionality anyway.
                                        > But of course it is easy for me to suggest this and that :)

                                        I always put more effort in functionality that works everywhere. And to
                                        keep programs portable rare features should be avoided, especially when
                                        they don't add something essential for the end user. MS thinks
                                        otherwise: They want programs to only run on MS-Windows. Portability
                                        means they might lose customers. Some Linux developers also go in this
                                        direction, they don't care about MS-Windows users. I care for everybody
                                        :-).

                                        --
                                        The process for understanding customers primarily involves sitting around with
                                        other marketing people and talking about what you would to if you were dumb
                                        enough to be a customer.
                                        (Scott Adams - The Dilbert principle)

                                        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                                        /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                                        \\\ download, build and distribute -- http://www.A-A-P.org ///
                                        \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                                        --~--~---------~--~----~------------~-------~--~----~
                                        You received this message from the "vim_dev" maillist.
                                        For more information, visit http://www.vim.org/maillist.php
                                        -~----------~----~----~----~------~----~------~--~---
                                      • Alexei Alexandrov
                                        ... Oops, yep, I have a lot of stuff to catch up with. -- Alexei Alexandrov --~--~---------~--~----~------------~-------~--~----~ You received this message
                                        Message 19 of 25 , Feb 6, 2008
                                        • 0 Attachment
                                          krischik wrote:
                                          >
                                          > I fear you missed something - since NTFS v3.0 named forks are
                                          > supported. They are called "Alternate data streams".
                                          >

                                          Oops, yep, I have a lot of stuff to catch up with.

                                          --
                                          Alexei Alexandrov


                                          --~--~---------~--~----~------------~-------~--~----~
                                          You received this message from the "vim_dev" maillist.
                                          For more information, visit http://www.vim.org/maillist.php
                                          -~----------~----~----~----~------~----~------~--~---
                                        Your message has been successfully submitted and would be delivered to recipients shortly.