You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Bo Berglund <bo...@gmail.com> on 2018/03/24 13:15:18 UTC
Subversion 1.9.7 server on Windows, advice on setting up svnsync?
I have an svn server running on a Windows Server 2016 with about 10
main repositories each with a number of projects.
I moved from CVS to svn 3 months ago.
On CVS I had a simple backup scheme where I used robocopy to nightly
copy out all changed files within the last 2 days and zip them up and
then send them to an offsite server.
This worked OK since CVS uses a 1:1 map of project files against the
reposirory repo files. So expanding the zips towards an initial copy
of the repo created a mirror.
Now I would like to do something similar for subversion and I have
found that there is a built-in svnsync command for this purpose.
Is there some current HOWTO I can use to set this up?
I mostly find old advice discussing improvements in upcoming versions
like 1.5 etc...
The plan is to use an Ubuntu server with the same version of svn
installed as the sync target. It would be connected via the Internet
through an OpenVPN tunnel or an SSL port on the target.
Some questions that have popped up:
- Should the nightly sync job run on the source or the target?
- Can the servers be using different OS (Win/Linux)?
- Should I use Apache or plain svn on the destination Linux server?
- I can port forward the svn server to the gateway on the target but
not on the source, does this affect where the job should run?
(If the target server is on a forwarded port then the OpenVPN tunnel
need not be set up, but the sync job must in that case run on the
Windows server source).
Advice welcome!
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Philip Martin <ph...@codematters.co.uk>.
Bo Berglund <bo...@gmail.com> writes:
> Good, then I will go ahead and erase the failed mirror repos and
> re-create them and then load the dump files. But for the not yet used
> repos I can just start loading the dumps. Saves some time.
There are some things you can do to make load faster. In 1.10 load has
a new --no-flush-to-disk option since a mirror typically doesn't need
the fsync() guarantees. On Linux you can use the eatmydata LD_PRELOAD
library to achieve as similar thing for releases before 1.10. You may
also find that increasing the memory cache speeds up loading:
eatmydata svnadmin load -M256 some-repo < some-dumpfile
--
Philip
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Thu, 29 Mar 2018 13:20:04 +0100, Philip Martin
<ph...@codematters.co.uk> wrote:
>Bo Berglund <bo...@gmail.com> writes:
>
>> 1) Do I have to disable the hooks I put in place for sync during the
>> load? Or will they not be triggered during a load operation?
>
>No, by default load does not invoke hooks. You can choose to invoke
>them via a command line options.
Good, then I will go ahead and erase the failed mirror repos and
re-create them and then load the dump files. But for the not yet used
repos I can just start loading the dumps. Saves some time.
>> 2) What do I do about the two repos where I have already run an
>> initialization command without the flag and where a number of
>> revisions have already been transferred before the error?
>
>If you want to load a dumpfile into these repositories then you need an
>incremental dumpfile that starts at the correct revision, not at zero.
>So to update a mirror where head is rN you need:
I don't need to continue from where they failed, it is easier for
these two to just load the full dump into a newly created empty repo.
>The master knows nothing about the mirror (except for any custom stuff
>that you put into the master hook files). All the svnsync
>initialization data is stored as revprops on r0 of the mirror. e.g:
That is the important item I needed! It means that if I kill the
existing partial mirror repos and re-create them anew I will have a
working target after I load the dump! :)
Thank you very much for your help! Much appreciated!
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Philip Martin <ph...@codematters.co.uk>.
Bo Berglund <bo...@gmail.com> writes:
> 1) Do I have to disable the hooks I put in place for sync during the
> load? Or will they not be triggered during a load operation?
No, by default load does not invoke hooks. You can choose to invoke
them via a command line options.
> 2) What do I do about the two repos where I have already run an
> initialization command without the flag and where a number of
> revisions have already been transferred before the error?
If you want to load a dumpfile into these repositories then you need an
incremental dumpfile that starts at the correct revision, not at zero.
So to update a mirror where head is rN you need:
svnadmin dump --incremental -rN+1:head master
> If I erase these repos and recreate them empty can I repeat the
> initialization command with the non-empty flag and it will work? Or is
> the fact that the initialization has been already executed without the
> flag on these two master repos place them in limbo?
> Can an svnsync initialize command be undone?
The master knows nothing about the mirror (except for any custom stuff
that you put into the master hook files). All the svnsync
initialization data is stored as revprops on r0 of the mirror. e.g:
$ svnlook proplist --verbose --revprop -r0 ../mirror/apache-repo-fsfs
svn:date
2011-08-04T21:40:25.140661Z
svn:sync-from-url
https://svn.apache.org/repos/asf/subversion
svn:sync-from-uuid
13f79535-47bb-0310-9956-ffa450edef68
svn:sync-last-merged-rev
1827963
You can modify/delete this data using propset/propdel commands.
--
Philip
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Thu, 29 Mar 2018 10:34:52 +0100, Philip Martin
<ph...@codematters.co.uk> wrote:
>Philip Martin <ph...@codematters.co.uk> writes:
>
>> Bo Berglund <bo...@gmail.com> writes:
>>
>>> 2) Use hotcopy to make a backup copy of everything including hooks.
>>> The caveat here is that the hotcopy needs to be using the exact same
>>> version on both master and mirror. But I have 1.9.7 master on Windows
>>> and 1.9.3 mirror on Ubuntu Linux
>>
>> No, hotcopy needs compatible versions. Any version of Subversion is
>> compatible with all older versions, and any 1.N.x is compatible with any
>> other 1.N.y. In your case 1.9.7 and 1.9.3 are compatible.
>
>It's a bit more complicated in practice. Any version of Subversion can
>read/write all the older repository formats and it is the repository
>format that needs to be compatible. So for the hotcopy to be useable on
>the mirror what matters is whether the version of Subversion on the
>mirror is compatible with the repository format on the master.
OK, but I have now created the dump files and transferred them to the
mirror server. On that I had prepared all of the repos for
synchronization and succeeded to do the initial sync on two of them
when I ran into the problem for the 3rd and 4th, which triggered this
last question round here.
Now I am in a situation that I could load the dump files for the repos
I have not tried to sync yet, thus I can use the --allow-non-empty
flag during initialization and get the sync started on a later
revision.
But I have two questions in this regard:
1) Do I have to disable the hooks I put in place for sync during the
load? Or will they not be triggered during a load operation?
2) What do I do about the two repos where I have already run an
initialization command without the flag and where a number of
revisions have already been transferred before the error?
If I erase these repos and recreate them empty can I repeat the
initialization command with the non-empty flag and it will work? Or is
the fact that the initialization has been already executed without the
flag on these two master repos place them in limbo?
Can an svnsync initialize command be undone?
I don't want to rename the mirror repo to something different...
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Philip Martin <ph...@codematters.co.uk>.
Philip Martin <ph...@codematters.co.uk> writes:
> Bo Berglund <bo...@gmail.com> writes:
>
>> 2) Use hotcopy to make a backup copy of everything including hooks.
>> The caveat here is that the hotcopy needs to be using the exact same
>> version on both master and mirror. But I have 1.9.7 master on Windows
>> and 1.9.3 mirror on Ubuntu Linux
>
> No, hotcopy needs compatible versions. Any version of Subversion is
> compatible with all older versions, and any 1.N.x is compatible with any
> other 1.N.y. In your case 1.9.7 and 1.9.3 are compatible.
It's a bit more complicated in practice. Any version of Subversion can
read/write all the older repository formats and it is the repository
format that needs to be compatible. So for the hotcopy to be useable on
the mirror what matters is whether the version of Subversion on the
mirror is compatible with the repository format on the master.
--
Philip
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Philip Martin <ph...@codematters.co.uk>.
Bo Berglund <bo...@gmail.com> writes:
> 2) Use hotcopy to make a backup copy of everything including hooks.
> The caveat here is that the hotcopy needs to be using the exact same
> version on both master and mirror. But I have 1.9.7 master on Windows
> and 1.9.3 mirror on Ubuntu Linux
No, hotcopy needs compatible versions. Any version of Subversion is
compatible with all older versions, and any 1.N.x is compatible with any
other 1.N.y. In your case 1.9.7 and 1.9.3 are compatible.
--
Philip
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Wed, 28 Mar 2018 12:41:49 -0500, Bo Berglund
<bo...@gmail.com> wrote:
>QUESTION:
>---------
>Is there a possibility to dump the source repos, then use the
>dumpfiles to set up the mirrors and finally tell svnsync to start
>syncing from the revision that is now the last in the mirror?
It seems like 3 options are described in the svnbook:
1) Initial data transfer using the synchronize function.
This is what is not really working in my situation.
2) Use hotcopy to make a backup copy of everything including hooks.
The caveat here is that the hotcopy needs to be using the exact same
version on both master and mirror. But I have 1.9.7 master on Windows
and 1.9.3 mirror on Ubuntu Linux
3) Use the dump + Load sequence to transfer the repo content.
Then after configuring all the hook scripts etc on the mirror the
synchronize init can be run with a flag to allow a non-empty target
repository.
Seems like I have to go route #3
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Wed, 28 Mar 2018 09:00:05 +0200, Johan Corveleyn
<jc...@gmail.com> wrote:
>> Transmitting file data ....... 8 long lines of dots
>> ......................svnsync: E120106: ra_serf: The server sent a
>> truncated HTTP response body.
>
>I'd start with going through the apache logs on both servers to look
>for more information about this error (since "The server sent a
>truncated HTTP response body." indicates to me that an error should be
>logged on that server).
Unfortunately it is not possible for me at the moment since I am
travelling...
>It's also possible that some other network component has truncated
>things ... proxies, security content filters, ...
I tested the synchronize function on two repositories and they both
triggered this error, but on different revs of course.
So I used svnlook on the master server to find out what was actually
being transferred for these revs. Turns out to be in both cases an
initial import of a big "library" file set.
In the case of the cmp repo it involved 9848 files some of which are
very big. In the other case it was also an import affecting 3669 files
of a cutdown version of the library..
The first import was done by cvs2svn when I converted the CVS repo to
SVN, but the other was an import I did when I created a repo for a
consultant who should have access only to a subset of our versioned
files. So I exported these and then imported the export into the new
repository.
So I guess that the sheer size of the imports it tries to sync is what
kills it, especially when doing it over the Internet....
QUESTION:
---------
Is there a possibility to dump the source repos, then use the
dumpfiles to set up the mirrors and finally tell svnsync to start
syncing from the revision that is now the last in the mirror?
It would certainly make it more controlled since I could ftp the dump
files from the source to the destination and then do the svn operation
locally with no network issues like latencies and such.
I have read that one must start with the target repo at rev 0, but
then I have also read some notes that later versions of svn have
improved on this....
We use 1.9.3 on the mirror server and 1.9.7 on the master server.
Grateful for any advice here!
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Mar 28, 2018 at 1:56 AM, Bo Berglund <bo...@gmail.com> wrote:
> I proceeded to sync our repositories. On the 3rd repo I get the
> following error at revision 210 (it contains 1290 revisions):
>
> E:\>svnsync synchronize https://home.mirrordomain.com/svn/cmp
> https://masterserver/svn/cmp
> Transmitting file data ....... 8 long lines of dots
> ......................svnsync: E120106: ra_serf: The server sent a
> truncated HTTP response body.
>
> The sync was interrupted so I tried restarting and it went directly to
> the Transmitting step as shown above.
> On my screen there were 8 lines of dots until the error appeared and
> it happened at *exactly* the same place for both the original and
> restarted sync command. Took a few mnutes to get there.
>
> What can this be caused by and what can I do to fix it?
>
> I am running the command on the master server (Windows Server 2016)
> towards an overseas server running Ubuntu 16.04
I'd start with going through the apache logs on both servers to look
for more information about this error (since "The server sent a
truncated HTTP response body." indicates to me that an error should be
logged on that server).
It's also possible that some other network component has truncated
things ... proxies, security content filters, ...
--
Johan
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
I proceeded to sync our repositories. On the 3rd repo I get the
following error at revision 210 (it contains 1290 revisions):
E:\>svnsync synchronize https://home.mirrordomain.com/svn/cmp
https://masterserver/svn/cmp
Transmitting file data ....... 8 long lines of dots
......................svnsync: E120106: ra_serf: The server sent a
truncated HTTP response body.
The sync was interrupted so I tried restarting and it went directly to
the Transmitting step as shown above.
On my screen there were 8 lines of dots until the error appeared and
it happened at *exactly* the same place for both the original and
restarted sync command. Took a few mnutes to get there.
What can this be caused by and what can I do to fix it?
I am running the command on the master server (Windows Server 2016)
towards an overseas server running Ubuntu 16.04
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Branko Čibej <br...@apache.org>.
On 27.03.2018 22:52, Bo Berglund wrote:
> When I move forward and prepare target repositories for syncing I need
> to activate two hooks in each repo (pre-revprop-change and
> start-commit). So there will be 18 scripts to create...
> Can I symlink the existing hooks into the new repositories so I only
> need to manage one set of files?
Yes, of course.
-- Brane
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Mon, 26 Mar 2018 08:22:16 -0500, Bo Berglund
<bo...@gmail.com> wrote:
>svn: E195019: Redirect cycle detected for URL
The error was caused by misconfiguration of the
/etc/apache2/mods-enabled/dav_svn.conf file following the instructions
in this webpage:
https://tecadmin.net/install-subversion-server-on-ubuntu/
A bogus line appeared in that instruction step 3:
Alias /svn /var/lib/svn <== This line must be removed
<Location /svn>
Now I have synced my first of 10 repositories on the main server.
It worked fine and took about 1 hour for 947 revisions.
Question:
---------
When I move forward and prepare target repositories for syncing I need
to activate two hooks in each repo (pre-revprop-change and
start-commit). So there will be 18 scripts to create...
Can I symlink the existing hooks into the new repositories so I only
need to manage one set of files?
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Mon, 26 Mar 2018 00:21:33 -0500, Bo Berglund
<bo...@gmail.com> wrote:
>Not knowing what to do next I tried the synchronize command:
>
>E:\>svnsync synchronize https://xxx.yyy.com/svn/bosse
>https://localserver/svn/bosse
>svnsync: E170011: Repository moved permanently to
>'https://xxx.yyy.com/svn/bosse'
>
>Apparently something has gone wrong, but what?
>And what does "moved permanently" mean?
>AFAICT nothing has moved or copied...
>
I tried to access the remote repository using Chrome from the master
server and it succeeded.
I got to the web view after I had told Chrome I wanted to go to the
site using an unverified certificate.
But then I tried to use the svn client on a command prompt and this
happened:
E:\>svn list https://xxx.yyy.com/svn/bosse
Redirecting to URL 'https://xxx.yyy.com/svn/bosse':
Redirecting to URL 'https://xxx.yyy.com/svn/bosse':
svn: E195019: Redirect cycle detected for URL
'https://xxx.yyy.com/svn/bosse'
What in the world does this really mean? Redirecting to itself does
not make any sense to me. Especially since it seems to work as
expected using a browser.
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Sat, 24 Mar 2018 22:40:55 -0500, Bo Berglund
<bo...@gmail.com> wrote:
Contrary to the previous message I found that the only way I could fix
the connectivity was to publish my backup Ubuntu server on the
Internet as a self-signed Apache service on port 443 on my Internet
router.
So the svnsync command will execute on the master server on Windows
and target the remote https URL.
I have read the svnbook 1.7 on the svnsync usage:
http://svnbook.red-bean.com/en/1.7/svn.reposadmin.maint.html#svn.reposadmin.maint.replication
and I have followed its instructions all along to the end.
So I have configured the target mirror server (Ubuntu 16.04) such that
it is accessible on a domain name via https and I have prepared the
hooks all according to the svnbook. The test repository sits at
revision 0.
When done on Ubuntu I moved over to the source server (Windows Server
2016 in a completely different location).
So on the Windows Server command line I issued the commands from the
svnbook.
First, initialize the sync - but I see errors (privacy data mangled):
E:\>svnsync initialize https://xxx.yyy.com/svn/bosse
https://localserver/svn/bosse --sync-username syncuser --sync-password
zzzzz
Error validating server certificate for 'https://xxx.yyy.com:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: xxx.yyy.com
- Valid: from Mar 25 00:52:45 2018 GMT until Mar 22 00:52:45 2028 GMT
- Issuer: xxx.yyy.com, IT, xxxsciences Inc, Jarfalla, Stockholm,
SE(user@somedomain.com)
- Fingerprint:
1C:84:47:97:89:7E:48:3B:C9:08:C0:F5:65:E8:01:B0:7F:DD:DB:21
(R)eject, accept (t)emporarily or accept (p)ermanently?
I have seen this before when accessing a selfsigned server for the
first time, so I replied p and got this:
svnsync: E170011: Repository moved permanently to
'https://xxx.yyy.com/svn/bosse'
What does this mean? I have *not* requested a move, just to initialize
syncronization of the source localserver to the destination mirror
xxx.yyy.com
Not knowing what to do next I tried the synchronize command:
E:\>svnsync synchronize https://xxx.yyy.com/svn/bosse
https://localserver/svn/bosse
svnsync: E170011: Repository moved permanently to
'https://xxx.yyy.com/svn/bosse'
Apparently something has gone wrong, but what?
And what does "moved permanently" mean?
AFAICT nothing has moved or copied...
If I google the visible error message not a single hit I get is
referring to command svnsync...
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Bo Berglund <bo...@gmail.com>.
On Sat, 24 Mar 2018 20:17:27 +0100, Andreas Stieger
<an...@gmx.de> wrote:
>On 03/24/2018 02:15 PM, Bo Berglund wrote:
>> Is there some current HOWTO I can use to set this up?
>
>http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.replication
>http://svnbook.red-bean.com/nightly/en/svn.serverconfig.httpd.html#svn.serverconfig.httpd.extra.writethruproxy
>
>> Some questions that have popped up:
>> - Should the nightly sync job run on the source or the target?
>
>Actually this could (and should) be a post-commit hook for an immediate
>sync.
>
>> - Can the servers be using different OS (Win/Linux)?
>
>Yes.
>
>> - Should I use Apache or plain svn on the destination Linux server?
>
>Both work, and this depends on your network behind it. Ensure that you
>have at least one layer of encryption and authentication.
I have just configured the Apache2 system on the remote Ubuntu server
to use SSL encryption (https access). I used this excellent howto:
https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-apache-in-ubuntu-16-04
So now I have one repository set up and accessible via https on my
domain. This has the same name as one of the repos on the production
server and now sits at revision 0 ready to test the svnsync from the
production server itself. Too late now to start any such testing, but
I will try it out tomorrow.
>> - I can port forward the svn server to the gateway on the target but
>> not on the source, does this affect where the job should run?
>
>Yes you can sync in this way, the sync can be run from any host
>including a third one (but I recommend server hooks). The sync can also
>be done via URLs (ports) deviating from the usual or declared sync target.
I plan to add hooks into the master server (VisualSVN with svn 1.9.7),
which is a Windows Server 2016. So I will have to translate the hook
scripts from shell language to batch. Seems best to use the
post-commit hook to sync at commit.
Do you know what syntax to use on Windows to have the same effect as
the & at the command line end on Linux?
--
Bo Berglund
Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
Posted by Andreas Stieger <an...@gmx.de>.
Hello,
On 03/24/2018 02:15 PM, Bo Berglund wrote:
> Is there some current HOWTO I can use to set this up?
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.replication
http://svnbook.red-bean.com/nightly/en/svn.serverconfig.httpd.html#svn.serverconfig.httpd.extra.writethruproxy
> Some questions that have popped up:
> - Should the nightly sync job run on the source or the target?
Actually this could (and should) be a post-commit hook for an immediate
sync.
> - Can the servers be using different OS (Win/Linux)?
Yes.
> - Should I use Apache or plain svn on the destination Linux server?
Both work, and this depends on your network behind it. Ensure that you
have at least one layer of encryption and authentication.
> - I can port forward the svn server to the gateway on the target but
> not on the source, does this affect where the job should run?
Yes you can sync in this way, the sync can be run from any host
including a third one (but I recommend server hooks). The sync can also
be done via URLs (ports) deviating from the usual or declared sync target.
Andreas