You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ru...@oxaion.de on 2006/02/06 11:22:48 UTC
using NTFS ADS, HFS+ ResourceForks or other file system metadata facility
instead of a .svn directory
I found a quite anonymous and sparse suggestion dated back in 2003. I would
like to elaborate on it and the objections raised against it. I have posted
most of this text to the users list, which was propably to the wrong
audience, but there was one very helpful answer from Angel Marin, which I
have assimilated.
NTFS can store arbitrary additional strams for a file or directory called
Alternate Data Streams ADS. E.g. you can do "echo foo > testfile:svn" (note
the colon) on an existing file "testfile" without changing its contents. To
retrieve that file fork do "more < testfile:svn". Editors using the
standard windows file open dialog can not open the ADS, but others like
jEdit can. And all of this even works on directories!!!
MacOS HFS+ supports only one additional file fork, called resource fork.
But the resource fork has (generally) a special structure so you can store
your own data using your own key (say ".svn") without disturbing and being
disturbed by others. It does *not* work on directories, though, and
somebody on this list stated that the use of the resource fork was
deprecated on MacOS X; so it might not be an option.
I do not know, if any of the common unix file systems support custom file
metadata, but I guess it would be at least in a planning stage.
All of these mechanisms had been introduced to store metadata. On Windows
it is used for storing file permission (ACL), the author, etc., so I guess
it lends itself as a natural place to store Subversion metadata as well,
rendering the .svn directory altogether dispensable, although not reducing
the disk space occupied. Even better: Moving, duplicating or renaming such
a file or directory would keep the metadata intact and attatched to the
file (Barry Scott had objected it would not, but at least on MacOS, Windows
2000 and XP it does, provided that the target filesystem is NTFS resp. HFS+
as well) . A whole bunch of problems working with versioned files without
subversion tools would simply disappear, including double matches in find
tools, etc.
Needless to say that the .svn directory would have to remain as a platform
independant alternative and fallback solution. It would be necessary have a
tool that can roll the .svn directory out into the file metadata and vice
versa (this should *not* happen automatically!).
Attatched custom file metadata would simplify most versioning use cases,
but not all:
1. Removing a file or directory would remove its meta data as well. In
order to know that something is missing, the directory would need to have a
list of known objects, so it can look for the same object with a different
name or at a different location (which still would have the old metadata)
in order to tell the repository to rename, move or even remove that object
(I do not think that we should have the user explicitly mark it as deleted
any more).
2. Moving or copying files or directories to a different file system would
*silently* remove the meta data. This happens already when you copy a
single file now, but not with directories. And the user might already have
got spoiled and not expect it ;-)
3. It is more difficult to *intentionally* remove the metadata. Maybe we
would need another tool for that.
How much code is involved in finding the correct path for the meta data for
a file or directory? Is the change to the existing code as easy/problematic
as it was renaming the directory to _svn?
There is another thing that might be done with this: Single file checkouts.
But as a lot of code might depend on only complete directories being
checked out, so this feature seems to be more risky to implement.
Regards
Rüdiger
Besuchen Sie uns auf der CeBIT in Hannover vom 09. - 15. März 2006.
Sie finden uns in Halle 5, Stand A38.
This message is intended for the addressee or its representative only.
Any form of unauthorized use, publication, reproduction, copying or
disclosure of the content of this e-mail is not permitted. If you are not
the intended recipient of this e-mail message and its contents, please
notify the sender immediately and delete this message and all its
attachments subsequently.
oxaion ag
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata facility instead of a .svn directory
Posted by David Rauschenbach <da...@goodserver.com>.
Joseph wrote:
> In general, I would suggest it might be appropriate to store
> a small amount of data as an ADS. For example, storing the
> subversion properties as an ADS sounds like it would be a
> sensible use (they might even stay MFT resident in which
> case you wouldn't incure a allocatition-block-size hit in
> disk space for a 12 byte string.)
I agree. And MFT-resident is what I was getting at when I mentioned locality. I guess each filesystem has a different amount of space in the i-node for an attribute map, or the first bytes anyway. NTFS: 12 bytes (sounds familiar from Helen Custer's book); BFS: 760 bytes (Practical Filesystem Design); HPFS: 0 (none of attributes stored with inode), with 64k upper bound. I don't know anything about Mac -- seems like I've been saying that for too many years now. But from what I've heard, the OS X team was the BFS team. Wikipedia's HFS+ entry ("6. Attributes File" section) seems to imply the answer is 0 (no attrs stored with inode). This page mentions a few interesting HFS+/xattr-API attribute naming and file naming rules.
Joseph wrote:
> Storing the textbase does not [make sense]-- both because of the large amount
> of space being 'hidden' and because you have no way to prevent
> the user from deleting it.
Again, sounds reasonable.
If the svn project were to follow such a tack, the .svn/props/ and .svn/prop-base/ folders could go away, which, among other benefits such as fixing OS metadata and search facilities, could have performance implications on certain platforms worth getting excited about.
David
----- Original Message -----
From: "Joseph Galbraith" <ga...@vandyke.com>
To: "David Rauschenbach" <da...@goodserver.com>
Cc: "Michael Sinz" <Mi...@sinz.org>; <de...@subversion.tigris.org>
Sent: Monday, February 06, 2006 1:01 PM
Subject: Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata facility instead of a .svn directory
> David Rauschenbach wrote:
>> Hi Michael,
>>
>> Michael wrote:
>>> I would much rather try
>>> sticking to the goal of Subversion being a revision management
>>> system and that the files are "sacred" and should not be corrupted
>>> or changed due to the use of Subversion. (I should get back what
>>> I put in)
>
> Just to kind of throw gasoline on the fire :-)
>
> NTFS doesn't allow an unlimited number of ADS per file. I don't
> recall what the limit is, but it was surprisingly low (at least
> I recall being surprised when I heard it-- several thousand but
> not 10's of thousands I think.)
>
> I'm pretty sure that ACLs are not really stored as a named data
> stream... so you couldn't overwrite the ACL with creative use
> of CreateFile("myfile:$ACL") or some such. Likewise, create
> time, change time, modified time, last access, and the file
> attributes (including archive bit) are also not stored as
> an ADS.
>
> In terms of 'sacred' data... I don't believe that it would
> have an impact. If subversions stored per-file meta data
> in a ADS call myfile.txt:.svn, you'd have the same chances
> of colliding with a real data stream as you do with today's
> .svn directory for meta-data.
>
> However, it was only as of Windows 2003 that microsoft
> finally introduced a real way to enumerate alternate
> data streams. (Previous to that you had to use tricks
> with the backup APIs and parsing the backup data.)
>
> As a result, most tools (including exploder) are not
> aware of streams. If you want to see and admin turn
> purple, mutter words like "root-kit", "malware", and
> other creative swear-words, take half his disk space
> and make it disappear from the view of all his available
> tools... it's gone, but he can't figure out where.
> Unfortunately, that is the state-of-ads as it stands
> today.
>
> In general, I would suggest it might be appropriate to store
> a small amount of data as an ADS. For example, storing the
> subversion properties as an ADS sounds like it would be a
> sensible use (they might even stay MFT resident in which
> case you wouldn't incure a allocatition-block-size hit in
> disk space for a 12 byte string.)
>
> Storing the textbase does not-- both because of the large amount
> of space being 'hidden' and because you have no way to prevent
> the user from deleting it.
>
> Thanks,
>
> Joseph
>
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata
facility instead of a .svn directory
Posted by Joseph Galbraith <ga...@vandyke.com>.
David Rauschenbach wrote:
> Hi Michael,
>
> Michael wrote:
>> I would much rather try
>> sticking to the goal of Subversion being a revision management
>> system and that the files are "sacred" and should not be corrupted
>> or changed due to the use of Subversion. (I should get back what
>> I put in)
Just to kind of throw gasoline on the fire :-)
NTFS doesn't allow an unlimited number of ADS per file. I don't
recall what the limit is, but it was surprisingly low (at least
I recall being surprised when I heard it-- several thousand but
not 10's of thousands I think.)
I'm pretty sure that ACLs are not really stored as a named data
stream... so you couldn't overwrite the ACL with creative use
of CreateFile("myfile:$ACL") or some such. Likewise, create
time, change time, modified time, last access, and the file
attributes (including archive bit) are also not stored as
an ADS.
In terms of 'sacred' data... I don't believe that it would
have an impact. If subversions stored per-file meta data
in a ADS call myfile.txt:.svn, you'd have the same chances
of colliding with a real data stream as you do with today's
.svn directory for meta-data.
However, it was only as of Windows 2003 that microsoft
finally introduced a real way to enumerate alternate
data streams. (Previous to that you had to use tricks
with the backup APIs and parsing the backup data.)
As a result, most tools (including exploder) are not
aware of streams. If you want to see and admin turn
purple, mutter words like "root-kit", "malware", and
other creative swear-words, take half his disk space
and make it disappear from the view of all his available
tools... it's gone, but he can't figure out where.
Unfortunately, that is the state-of-ads as it stands
today.
In general, I would suggest it might be appropriate to store
a small amount of data as an ADS. For example, storing the
subversion properties as an ADS sounds like it would be a
sensible use (they might even stay MFT resident in which
case you wouldn't incure a allocatition-block-size hit in
disk space for a 12 byte string.)
Storing the textbase does not-- both because of the large amount
of space being 'hidden' and because you have no way to prevent
the user from deleting it.
Thanks,
Joseph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata
facility instead of a .svn directory
Posted by Michael Sinz <Mi...@sinz.org>.
David Rauschenbach wrote:
> Hi Michael,
>
> Michael wrote:
>> I would much rather try
>> sticking to the goal of Subversion being a revision management
>> system and that the files are "sacred" and should not be corrupted
>> or changed due to the use of Subversion. (I should get back what
>> I put in)
>
> I have not heard any suggestions yet that would challenge subversion's
> ability to be an uncompromised revision management system. I can't see
> why anyone would want that. Understood if you mean the data stream and
> probably created-time and modified-time streams are sacred. Those
> certainly play into things you might want to "get back" just the way you
> put them in.
>
> However, are other streams really sacred? Do you have strong feelings
> about how the archive bit might be set on an NTFS file of a server
> repository? If not, then I don't see why you'd have strong feelings
> about other potential streams, none of which jeopardize the sacred streams.
There are many other streams that are important. Some development tools
store tag information in the xattr/forks. The Mac stores resource data
and compiled plist stuff (why they went back to forks for that is beyond
me, but they did)
So, there is more than just the simple time/date/etc stuff here.
> Similarly, I don't see why svn would or should care about non-special
> properties (not in the "svn:" namespace).
>
> Also, beyond asking those questions, I should add my opinion into the
> fray! The .svn folders *doubles* the number of folders that live on my
> hard drive, and the repository server's hard drive. It slows everything
> down, even backup/restore of all the workstations. Not to mention that
> when alternate streams are small, there is the lost benefit of stored
> locality. Plus, from a usability standpoint, it's not long before I end
> up browsing folders in a way that needs to include hidden folders, and
> the .svn folders clutter up everything.
Yes, this is a problem. But think of the .svn folder as its own "stream"
that is associated with the current directory (the containing directory)
Once you look at it like that, then the .svn directories are really just
containers for that data and if/when a client has a filesystem that can
handle that behavior better, it should be able to do so.
Think of a filesystem that is databased based like BeOS or the future
WinFS stuff from Microsoft - the possibilities get really interesting.
The problem is that not all filesystems are like that and not all users
use them so you end up with the .svn directories.
(BTW - I (and others at C=) were pushing hard to make our applications
really be folders and thus having multiple "streams" in the application
was just really multiple files in the folder. That is what NeXT and now
Mac OS-X do for applications and it works really, really well - until you
use a command line and "cd" around your disk to see what it really looks
like.)
> It's noise when metadata shows up as files.
:-) Then fix your file browser :-) I think it is bad when you can not
see that there is metadata in the system using normal tools. (Show me
where Windows can display you the ADS from NTFS...)
At least with hidden/special directories they can be hidden or not as
per the user's choice.
> As these "object filesystems" (NT terminology) get built out, and once
> people start relying on them, they will realize that all metadata *not*
> stored correctly becomes an impedance mismatch. For example, file
> traversal searches and desktop search engine searches getting false-hits
> from a metadata file that is actually "metadata plus pointer" to a real
> file, but the search tool doesn't and shouldn't know that. Similarly,
> BeOS/Haiku indexes unable to index metadata for active search or passive
> search (categorization), because the metadata is "misfiled". Which
> ultimately causes adapters like Google desktop search plug-ins to get
> the work done that could have been done correctly in the first place.
>
> If the current svn filesystem abstraction is an impedance mismatch with
> the reality of emerging filesystems, then that's only going to become a
> bigger problem over time, as subversion becomes more successful (yahoo!)
> and metadata storage starts getting corrected by the industry as a
> whole. The industry has spent big money over more than a decade getting
> this done... and not without good reason. No point in even digressing
> into that -- the filesystems are here now, and how they got here is
> already history. Back at the high level, it seems to me like if this
> doesn't get addressed now, it will still have to get addressed later.
I don't think it is a mismatch - I just think that the current enhanced
filesystems are not enhanced enough and the tools are too crude as far
as dealing with the enhancements. Lets see where/how that goes in the
future, but remember there is a very long adoption curve for these types
of ideas. (Mac started with this in 1984 and the Lisa in 1982 - so this
is not new stuff and yet it still is not mainstream in any real way)
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:michael.sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata facility instead of a .svn directory
Posted by David Rauschenbach <da...@goodserver.com>.
Hi Michael,
Michael wrote:
> I would much rather try
> sticking to the goal of Subversion being a revision management
> system and that the files are "sacred" and should not be corrupted
> or changed due to the use of Subversion. (I should get back what
> I put in)
I have not heard any suggestions yet that would challenge subversion's
ability to be an uncompromised revision management system. I can't see why
anyone would want that. Understood if you mean the data stream and probably
created-time and modified-time streams are sacred. Those certainly play into
things you might want to "get back" just the way you put them in.
However, are other streams really sacred? Do you have strong feelings about
how the archive bit might be set on an NTFS file of a server repository? If
not, then I don't see why you'd have strong feelings about other potential
streams, none of which jeopardize the sacred streams.
Similarly, I don't see why svn would or should care about non-special
properties (not in the "svn:" namespace).
Also, beyond asking those questions, I should add my opinion into the fray!
The .svn folders *doubles* the number of folders that live on my hard drive,
and the repository server's hard drive. It slows everything down, even
backup/restore of all the workstations. Not to mention that when alternate
streams are small, there is the lost benefit of stored locality. Plus, from
a usability standpoint, it's not long before I end up browsing folders in a
way that needs to include hidden folders, and the .svn folders clutter up
everything.
It's noise when metadata shows up as files.
As these "object filesystems" (NT terminology) get built out, and once
people start relying on them, they will realize that all metadata *not*
stored correctly becomes an impedance mismatch. For example, file traversal
searches and desktop search engine searches getting false-hits from a
metadata file that is actually "metadata plus pointer" to a real file, but
the search tool doesn't and shouldn't know that. Similarly, BeOS/Haiku
indexes unable to index metadata for active search or passive search
(categorization), because the metadata is "misfiled". Which ultimately
causes adapters like Google desktop search plug-ins to get the work done
that could have been done correctly in the first place.
If the current svn filesystem abstraction is an impedance mismatch with the
reality of emerging filesystems, then that's only going to become a bigger
problem over time, as subversion becomes more successful (yahoo!) and
metadata storage starts getting corrected by the industry as a whole. The
industry has spent big money over more than a decade getting this done...
and not without good reason. No point in even digressing into that -- the
filesystems are here now, and how they got here is already history. Back at
the high level, it seems to me like if this doesn't get addressed now, it
will still have to get addressed later.
David
----- Original Message -----
From: "Michael Sinz" <Mi...@sinz.org>
To: <de...@subversion.tigris.org>
Sent: Monday, February 06, 2006 9:53 AM
Subject: Re: using NTFS ADS, HFS+ ResourceForks or other file system
metadata facility instead of a .svn directory
On 2/6/06, Julian Foad <ju...@btopenworld.com> wrote:
> Ruediger.Dohna@oxaion.de wrote:
> [...]
> > it lends itself as a natural place to store Subversion metadata as well,
> > rendering the .svn directory altogether dispensable, [...]
>
> This is an interesting idea. There are two levels at which we could use
> filesystem metadata storage:
>
> * just for storing the Subversion "properties" on an item;
>
> * for storing all Subversion meta-data that is currently in the ".svn"
> directory - pristine (base) copies, information about which
> repository/path/revision the local files are connected to, etc.
>
> You seem to be talking about the latter.
>
> Of course a fair bit of work would be required to do this, so it would
> need to
> have some distinct advantages.
[...]
> I can't yet see a compelling reason to use filesystem metadata storage.
I can think of at least one more reason not use the filesystem
metadata for Subversion - the files I am putting into the repository
have metadata for their own use. As such, modifying or adding
to this changes the file I have placed into the repository.
(Note - currently, Subversion does not actually deal with metadata
within files very well [ok - at all :-)] which is a whole different
problem and also non-trivial to fix in a cross-platform way)
There are other tricky issues that start to come up, but I would like
to not go down those rabbit holes now. I would much rather try
sticking to the goal of Subversion being a revision management
system and that the files are "sacred" and should not be corrupted
or changed due to the use of Subversion. (I should get back what
I put in) [Yes, I know that we don't do metadata or xattrs yet]
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:Michael.Sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata facility instead of a .svn directory
Posted by Michael Sinz <Mi...@sinz.org>.
On 2/6/06, Julian Foad <ju...@btopenworld.com> wrote:
> Ruediger.Dohna@oxaion.de wrote:
> [...]
> > it lends itself as a natural place to store Subversion metadata as well,
> > rendering the .svn directory altogether dispensable, [...]
>
> This is an interesting idea. There are two levels at which we could use
> filesystem metadata storage:
>
> * just for storing the Subversion "properties" on an item;
>
> * for storing all Subversion meta-data that is currently in the ".svn"
> directory - pristine (base) copies, information about which
> repository/path/revision the local files are connected to, etc.
>
> You seem to be talking about the latter.
>
> Of course a fair bit of work would be required to do this, so it would need to
> have some distinct advantages.
[...]
> I can't yet see a compelling reason to use filesystem metadata storage.
I can think of at least one more reason not use the filesystem
metadata for Subversion - the files I am putting into the repository
have metadata for their own use. As such, modifying or adding
to this changes the file I have placed into the repository.
(Note - currently, Subversion does not actually deal with metadata
within files very well [ok - at all :-)] which is a whole different
problem and also non-trivial to fix in a cross-platform way)
There are other tricky issues that start to come up, but I would like
to not go down those rabbit holes now. I would much rather try
sticking to the goal of Subversion being a revision management
system and that the files are "sacred" and should not be corrupted
or changed due to the use of Subversion. (I should get back what
I put in) [Yes, I know that we don't do metadata or xattrs yet]
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:Michael.Sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata
facility instead of a .svn directory
Posted by Julian Foad <ju...@btopenworld.com>.
Ruediger.Dohna@oxaion.de wrote:
[...]
> it lends itself as a natural place to store Subversion metadata as well,
> rendering the .svn directory altogether dispensable, [...]
This is an interesting idea. There are two levels at which we could use
filesystem metadata storage:
* just for storing the Subversion "properties" on an item;
* for storing all Subversion meta-data that is currently in the ".svn"
directory - pristine (base) copies, information about which
repository/path/revision the local files are connected to, etc.
You seem to be talking about the latter.
Of course a fair bit of work would be required to do this, so it would need to
have some distinct advantages.
> Even better: Moving, duplicating or renaming such
> a file or directory would keep the metadata intact and attatched to the
> file [...]
That is currently true for a directory but not for a file. Below, you suggest
that Subversion might automatically find the new item and save you from having
to tell it that you moved it, but I don't think that's practical. Can you give
any other example of a situation in which it would be useful for a file to keep
its metadata with it?
> A whole bunch of problems working with versioned files without
> subversion tools would simply disappear, including double matches in find
> tools, etc.
Avoiding double matches in find tools is one advantage; that problem still
annoys me fairly frequently. Can you list some other advantages?
[Disadvantages]
> 1. Removing a file or directory would remove its meta data as well. In
> order to know that something is missing, the directory would need to have a
> list of known objects,
Yes. Also, if we still wanted to allow a deleted file to be reverted
(restored) without network access, we would need to keep the file's pristine
(base) text in the directory's metadata.
> so it can look for the same object with a different
> name or at a different location (which still would have the old metadata)
> in order to tell the repository to rename, move or even remove that object
> (I do not think that we should have the user explicitly mark it as deleted
> any more).
Here you are talking about a different mode of operation, a mode in which
Subversion acts as if an "svn delete" command has been issued if the local file
is gone, or "svn move" if the file has been moved, and so on. (In order to do
things like make a local back-up of the file, and avoid Subversion adding it to
the repository, we would also need a command to tell Subversion to forget about
a particular file while leaving it on disk.)
a) Could Subversion really, practically, look for an object that has
disappeared (been moved/renamed/deleted)? Assuming it could identify the
object (with a unique ID, say) if it found it, where would it look? I think it
would have to look "everywhere", which is not practical. Sometimes it might
find it quickly, nearby, but if deleted it would have to search the whole file
system. The only practical way I can see is for Subversion to hook into the
file system and be notified of move/rename/delete operations. A program
operating in that way would be a completely different sort of Subversion client
from what we have now.
b) How would the use of filesystem meta-data storage facilitate this mode of
operation? OK, it would make it easier to identify whether and how a
particular file is connected to Subversion, but it doesn't fundamentally make
it possible. It seems to be a completely separate issue.
I can't yet see a compelling reason to use filesystem metadata storage.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: using NTFS ADS, HFS+ ResourceForks or other file system metadata
facility instead of a .svn directory
Posted by Martin Furter <mf...@rola.ch>.
On Mon, 6 Feb 2006 Ruediger.Dohna@oxaion.de wrote:
> 3. It is more difficult to *intentionally* remove the metadata. Maybe we
> would need another tool for that.
I guess it's much easier to remove the metadat accidently.
If an application (editor or whatever) writes the data to a temporary file
first and then moves it in place the metadata will probably be lost.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org