You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Timothee Besset <ti...@qeradiant.com> on 2001/10/01 12:06:31 UTC
svn and "mostly-binary" repositories
I got some time to download subversion, recompile it and wander through
the docs. No doubt that SVN is gonna be a great replacement and
enhancement to CVS. I am particularly interested in an area where CVS
isn't doing great: binary file repositories.
Let's say you have a project that deals with a lot of multimedia data such
as sound files, images, and other monolitic binary objects. And you want
to allow collaborative work with those files to a bunch of people.
Being long time CVS users, our first solution was to use it. But CVS gets
insanely slow when the repository gets big. I haven't clearly identified
what's happening yet, it seems that on a cvs commit it will try to fetch
all the repository files to see if they have been modified before
performing the commit. This is going to be a blocking situation for us
pretty soon, given that our target repository size is >100Mb
I have looked for open source / free software alternatives that can
provide a solution for binary repositories but didn't find any satisfying
solution. So I am looking at SVN now. If you guys know some interesting
solution, please send me the links.
Does the design of SVN have some specific features to support binary
repositories? I will try to list below the major points when dealing with
binary repositories:
- Efficient file transfer protocols. Using rsync might be of interest.
- Efficient file comparison. When deciding wether a local file is in sync
with the repository, use md5sum instead of fetching and diffing.
- versions and branches as usual in CVS.
- file locking. given that binary files don't merge, concurrency has a
reduced importance. File locking abilities would be a great bonus. Either
the intrusive way, like setting the repository to readonly and having
to specifically lock the files you are going to change, or non-intrusive
with a daemon locking the files you modify and keeping readonly the files
locked by others.
If I can just get those 4 features working, I'd be very very happy :-)
Now I'm waiting for opinions of subversions gurus wether this is part of
SVN's objectives, wether there has been some design done around that and
wether they think it would be worth doing...
regards
TTimo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: "mostly-binary" repositories, conclusion
Posted by Ben Collins-Sussman <su...@collab.net>.
Timothee Besset <ti...@qeradiant.com> writes:
> Ok .. thanks for all the feedback. As a conclusion, the main question I'd
> like to ask: do you think there is an interest for me to setup an
> experimental SVN server and maintain our binary repository there?
I'd wait a little bit longer. At this exact moment, our network layer
is in fact *not* always sending diffs in both directions, even though
it's theoretically possible. It's an optimization that will be added
soon. Before you start using a repository of huge binary files, I'd
wait for this optimization. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: "mostly-binary" repositories, conclusion
Posted by kf...@collab.net.
Timothee Besset <ti...@qeradiant.com> writes:
> Ok .. thanks for all the feedback. As a conclusion, the main question I'd
> like to ask: do you think there is an interest for me to setup an
> experimental SVN server and maintain our binary repository there?
There's benefit for us, but not for you. :-)
We'll get valuable debugging information. You'll get to... well,
you'll get to send in valuable debugging information :-). I highly
doubt that your users would benefit until later this fall, though.
It's not ready for production use, except on itself.
-K
> I can see several issues:
>
> - SVN client. Precompiled binaries would be good .. maybe not unix ones,
> but win32 and OSX. Command line should be good enough.
>
> - Setup and maintenance of a server. The docs seem fairly complete about
> that.
>
> Is it really gonna make any difference with a CVS-hosted repository? (I
> guess I need to setup a server and see by myself).
>
> regards
>
> TTimo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
"mostly-binary" repositories, conclusion
Posted by Timothee Besset <ti...@qeradiant.com>.
Ok .. thanks for all the feedback. As a conclusion, the main question I'd
like to ask: do you think there is an interest for me to setup an
experimental SVN server and maintain our binary repository there?
I can see several issues:
- SVN client. Precompiled binaries would be good .. maybe not unix ones,
but win32 and OSX. Command line should be good enough.
- Setup and maintenance of a server. The docs seem fairly complete about
that.
Is it really gonna make any difference with a CVS-hosted repository? (I
guess I need to setup a server and see by myself).
regards
TTimo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svn and "mostly-binary" repositories
Posted by Ben Collins-Sussman <su...@collab.net>.
Timothee Besset <ti...@qeradiant.com> writes:
> The main problem I think is how to decide wether the local binary file is
> different from the repository one. Implementing a comparison by hashing
> the file would make sense I think, since it doesn't make a lot of sense to
> apply the patching techniques of svn to binary files.
As Greg said, earlier, Subversion currently caches secret, "pristine"
copies of files in the SVN/ area. It lets us do nice things, like
send diffs from client to server, run 'svn diff' and 'svn revert' as
non-network commands, etc.
The working-copy library uses this algorithm to detect if any file is
locally modified:
* if the timestamps between the working and "pristine" files are
identical, return NOT_CHANGED
* else if the filesizes are different, return CHANGED
* else do a byte-for-byte brute force comparison, and return
result.
All files are treated equally in this regard; binari-ness or
text-ness doesn't matter.
If you commit a locally-modified file, a binary diff algorithm is
always used to transmit changes. It's probably not as tricky as
rsync, but it seems to be working well.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svn and "mostly-binary" repositories
Posted by Timothee Besset <ti...@qeradiant.com>.
My suggestion about rsync was only a part of the problem. Optimizing the
transfer of the binary file once it has been detected that they are
different is only a second step. And after all it's not that important
since individual binary files we are working on never get really big, so
they could go through HTTP just fine.
The main problem I think is how to decide wether the local binary file is
different from the repository one. Implementing a comparison by hashing
the file would make sense I think, since it doesn't make a lot of sense to
apply the patching techniques of svn to binary files.
TTimo
On Mon, 1 Oct 2001 18:23:14 +0200
>
> I think the rsync protocol might have some place in svn, but
> I don't think anyone is going to look into that until after 1.0.
>
> > Btw, there's a librsync that might be useful for subversion.
>
> Yes, but it is YAL (Yet Another Library), and the concept of
> rsync is much more important than the implementation asis in
> librsync IMHO.
>
> Sander
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: svn and "mostly-binary" repositories
Posted by Sander Striker <st...@apache.org>.
> > > - Efficient file comparison. When deciding wether a local
> > > file is in sync with the repository, use md5sum instead of
> > > fetching and diffing.
> >
> > What we do at the moment is store a pristine copy of each
> > working copy file in the admin directory. This allows us to
> > send diffs across the wire for commits, and to determine
> > whether a file has been modified by looking at mod times and
> > performing a local diff if they're different. Unfortunately,
> > it also comes at a 100% space penalty in the working copy; a
> > checkout of a 1GB source directory will come in at 2GB.
> >
> > In theory, we will make it an option in some post-1.0 release
> > to omit the pristine copy in the working copy, and instead use
> > hash checksums of some kind to decide whether a local file is
> > in sync with the repository. And, of course, to do a commit
> > the client would have to send the new complete text rather
> > than send a diff.
> [snip]
>
> As I understand it, rsync works basically like so:
>
> The server and the client both chop the file into blocks. They
> both then compute checksums of all the blocks. One then sends
> its list of block checksums to the other. Then the blocks that
> are different are transferred.
>
> i.e. it's more efficient than just transferring the whole file,
> and you don't need to keep a copy of the original to see what's
> changed.
That is not really an accurate description of rsync.
- The client chops the local file into equal sized blocks
and calculates 2 checksums, md4 and adler32, it sends
those to the server
- The server moves a block sized window over the file
on the server side and calculates the adler32 checksum
for each offset. When an adler32 match occurs with the
client, the md4 hash is compared. On a match, the index
off the block is noted as a token and the window shifts an
entire block size. If there was no match, the window shifts
one byte, noting the current byte. All this data is sent
to the client.
- The client recreates the file by concat'ing blocks (identified
by the tokens, which can be simple indexes in the checksum
table) and bytes.
That's it. The reason why rsync is a pretty complex piece of
code is because directory trees are rsync'd over aswell, and the
network protocol is also squeezed to using every bit available.
That is the reason why the rsync network protocol was never
presented as an RFC.
I think the rsync protocol might have some place in svn, but
I don't think anyone is going to look into that until after 1.0.
> Btw, there's a librsync that might be useful for subversion.
Yes, but it is YAL (Yet Another Library), and the concept of
rsync is much more important than the implementation asis in
librsync IMHO.
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svn and "mostly-binary" repositories
Posted by Michael Wood <mw...@its.uct.ac.za>.
On Mon, Oct 01, 2001 at 08:25:33AM -0400, Greg Hudson wrote:
> > - Efficient file transfer protocols. Using rsync might be of
> > interest.
>
> I'm not sure how rsync would fit. We are definitely much more
> sparing with bytes on the wire than CVS is, though; we always
> send diffs where possible, never the complete text of files.
>
> > - Efficient file comparison. When deciding wether a local
> > file is in sync with the repository, use md5sum instead of
> > fetching and diffing.
>
> What we do at the moment is store a pristine copy of each
> working copy file in the admin directory. This allows us to
> send diffs across the wire for commits, and to determine
> whether a file has been modified by looking at mod times and
> performing a local diff if they're different. Unfortunately,
> it also comes at a 100% space penalty in the working copy; a
> checkout of a 1GB source directory will come in at 2GB.
>
> In theory, we will make it an option in some post-1.0 release
> to omit the pristine copy in the working copy, and instead use
> hash checksums of some kind to decide whether a local file is
> in sync with the repository. And, of course, to do a commit
> the client would have to send the new complete text rather
> than send a diff.
[snip]
As I understand it, rsync works basically like so:
The server and the client both chop the file into blocks. They
both then compute checksums of all the blocks. One then sends
its list of block checksums to the other. Then the blocks that
are different are transferred.
i.e. it's more efficient than just transferring the whole file,
and you don't need to keep a copy of the original to see what's
changed.
Btw, there's a librsync that might be useful for subversion.
--
Michael Wood
<mw...@its.uct.ac.za>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org