You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Shahaf <d....@daniel.shahaf.name> on 2012/08/09 13:25:44 UTC
rsyncability of fsfs f6
The question of keeping a FSFS mirror repository using rsync came up.
For FSFS f4 it's possible to do an rsync of just the rev file + revprop
file + 'current' from post-commit. (Yes, the prior revisions must be a
byte-for-byte match, and min-unpacked-rev must agree as well.
As to rep-cache.db, it is not needed on the mirror.)
In short, for f4 it seems to be a pretty efficient way of sync, although
of course it comes without forward-compatibility guarantees (i.e., it
has the acceptable-to-our-use-case caveat that it might stop working
come 1.8 or 1.9).
Does this property hold for f6 / 1.8?
Re: rsyncability of fsfs f6
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Stefan Fuhrmann wrote on Fri, Aug 10, 2012 at 13:27:59 +0200:
> On Thu, Aug 9, 2012 at 1:28 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > Daniel Shahaf wrote on Thu, Aug 09, 2012 at 12:25:44 +0100:
> >> Does this property hold for f6 / 1.8?
> >
> > It seems to me it would work via one of two methods,
> >
> > 1) rsync --delete the $rev.* files and then reconstruct the manifest on
> > the target;
>
> Not sure what rsync will do when copying a directory.
> Assume the following sequence
>
> * rsync reads list of files in folder
> * svn propset splits a revprop pack file
> * rsync might not copy the new files but won't find the old pack file
> -> data loss?
>
> If rsync re-tries the process in that case, we won't lose data
> but must remove new pack pack files manually, if an old one
> exists. That is because it is hard to ensure that *all* new files
> have already been written.
>
Yes, I realised that post-sending: we should copy all revprop files and
then manually delete old ones on the target.
> > 2) exclude revprops/ from the rsync and just use 'svnsync copy-revprops'
> > from post-commit and post-revprop-change.
> >
> > Other options?
>
> * snapshots, if supported by the OS / file system
> * keeping the lock file open, i.e. prevent writes while rsync'ing
> (maybe during a second sync run only)
>
Good ideas, thanks.
So the bottom line is: f6 revprops are _not_ rsync'able without
additional code, but there are ways to overcome this.
> -- Stefan^2.
>
> --
> Join us this October for Subversion Live 2012 – 2 full days of
> training, networking, live demos and more! 25% off before Aug. 10th
> with discount code “earlybird.”
>
> Certified & Supported Apache Subversion Downloads:
> http://www.wandisco.com/subversion/download
Re: rsyncability of fsfs f6
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Thu, Aug 9, 2012 at 1:28 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Daniel Shahaf wrote on Thu, Aug 09, 2012 at 12:25:44 +0100:
>> Does this property hold for f6 / 1.8?
>
> It seems to me it would work via one of two methods,
>
> 1) rsync --delete the $rev.* files and then reconstruct the manifest on
> the target;
Not sure what rsync will do when copying a directory.
Assume the following sequence
* rsync reads list of files in folder
* svn propset splits a revprop pack file
* rsync might not copy the new files but won't find the old pack file
-> data loss?
If rsync re-tries the process in that case, we won't lose data
but must remove new pack pack files manually, if an old one
exists. That is because it is hard to ensure that *all* new files
have already been written.
> 2) exclude revprops/ from the rsync and just use 'svnsync copy-revprops'
> from post-commit and post-revprop-change.
>
> Other options?
* snapshots, if supported by the OS / file system
* keeping the lock file open, i.e. prevent writes while rsync'ing
(maybe during a second sync run only)
-- Stefan^2.
--
Join us this October for Subversion Live 2012 – 2 full days of
training, networking, live demos and more! 25% off before Aug. 10th
with discount code “earlybird.”
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: rsyncability of fsfs f6
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Shahaf wrote on Thu, Aug 09, 2012 at 12:25:44 +0100:
> Does this property hold for f6 / 1.8?
It seems to me it would work via one of two methods,
1) rsync --delete the $rev.* files and then reconstruct the manifest on
the target;
2) exclude revprops/ from the rsync and just use 'svnsync copy-revprops'
from post-commit and post-revprop-change.
Other options?
Daniel
(the reason we consider rsync in the first place is because it might be
faster, and because one of our use-cases is to copy a repository to a
NAS box that has rsync installed but doesn't have svn installed.)