You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Tony Sweeney <sw...@addr.com> on 2015/02/20 08:54:16 UTC
svnsync problems
Hi,
Mise en scene:
We have three Subversion servers, version 1.6.11 running on CentOS6.
One is the live master and and the others are hot standbys in different
datacenters. The intent is that these are kept in sync with the master
using svnsync from a post-commit hook. One of the slaves is already set
up this way and has been since it was deployed by one of my predecessors
at this company. This is syncing nicely. The other slave instance was
also originally set up this way but was then modified to instead use a
block level filesystem mirroring technology, again by one
of my predecessors. We have since concluded that the mirroringsolution
was not sufficiently trustworthy, and so we would like to put it back to
the svnsync method that it had run in the past. The live master has in
excess of 366,000 revisions and new revisions come in at ~500 commits/day.
One solution would be to simply create an empty repo, set the necessary
r0 revprops and hook scripts and sync from scratch. However, we
estimate that this would take 10-15 days due to the size of the
repository, so we would like to short-circuit this by starting from a
hotcopy. We take hotcopies nightly, save them to a backup server and
'svnadmin verify' them there. I am trying to bring this slave back on
sync using the method described here:
http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
But using a hotcopy rather than a dump.
The master hotcopy is restored in the correct location on the slave and
the hook scripts fixed up using Puppet (master and slave necessarily
have different hook script setups). Permissions are fixed up to reflect
that the hotcopy is taken as root but the server is running under
Apache. I set the svn:sync-last-merged-rev revprop on r0 to reflect
the newest revision on the hotcopy. At this point (or so I thought) I
should be able to replicate over the changes since last night by running
svnsync manually and then enable it in the hook script.
The problem is that that the revisions seem to be corrupted on transfer
to this slave.
From the master:
[root@uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive
--trust-server-
cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
--sync-username svnsync --sync-password XXXXXXXX --source-username
svnsync --source-password XXXXXXXX --config-dir
/repos/svn/yoyodyne/hooks/certs
Transmitting file data ..
Committed revision 367219.
Copied properties for revision 367219.
Transmitting file data .
Committed revision 367220.
Copied properties for revision 367220.
Transmitting file data .
Committed revision 367221.
Copied properties for revision 367221.
Transmitting file data .svnsync: Corrupt representation '367220 0 41
29949930e5
5f203ff7027917e765e6ca7d
2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
You have new mail in /var/spool/mail/root
[root@uk-svn-1-new yoyodyne]#
In fact, checking on the slave, they have all come across corrupt:
[root@uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
* Verified revision 367200.
* Verified revision 367201.
* Verified revision 367202.
* Verified revision 367203.
* Verified revision 367204.
* Verified revision 367205.
* Verified revision 367206.
* Verified revision 367207.
* Verified revision 367208.
* Verified revision 367209.
* Verified revision 367210.
* Verified revision 367211.
* Verified revision 367212.
* Verified revision 367213.
* Verified revision 367214.
* Verified revision 367215.
* Verified revision 367216.
* Verified revision 367217.
* Verified revision 367218.
svnadmin: Corrupt representation '367219 0 50
633695f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
svnadmin: Malformed representation header
You have mail in /var/spool/mail/root
[root@uk-svn-2-new svn]#
Both the master and working slave verify all revisions clean at all
times. I've tried multiple hotcopies and every time the first revision
that it tries to sync comes across as corrupt.
On the master the revision file looks as follows:
[root@uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults =
trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnq/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnq/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
[root@uk-svn-1-new yoyodyne]#
The working slave looks as follows:
[root@us-svn-1 yoyodyne]# head -30 db/revs/367/367219
DELTA 366873 14907 1619
SVN??@
?=??h?&ng serialVersionUID = 1;
ENDREP
DELTA 366873 16539 1063
SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults =
trueif(results.size() <
requiredResults){
hasMoreResults = false;
}
hasMoreResults;
}
}
ENDREP
id: 1j-366873.0-228442.r367219/295
type: file
pred: 1j-366873.0-228442.r366873/30631
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7yyk/_e
props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/692
type: file
pred: 1h-366873.0-228442.r366873/32680
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7yyk/_d
props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
edRowList.java
copyroot: 228442 /MIME/MADb
[root@us-svn-1 yoyodyne]#
But on the failing slave:
[root@uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
id: 1j-366873.0-228442.r367219/0
type: file
pred: 1j-366873.0-228442.r366873/30632
count: 1
text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c
dc0cc36f718b468fc08817
88900f096d5cd4913f 367218-7wnt/_e
props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
copyroot: 228442 /MIME/MADb
id: 1h-366873.0-228442.r367219/395
type: file
pred: 1h-366873.0-228442.r366873/32681
count: 1
text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c
1f15b1d8cea246d731676f43
b2df4c74a1710793 367218-7wnt/_d
props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
cpath:
/MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
copyroot: 228442 /MIME/MADb
PLAIN
K 19
CSVQueryThread.java
V 37
file 1e-366873.0-228442.r366873/31862
K 18
CachedRowList.java
V 35
file 1h-366873.0-228442.r367219/395
K 25
DBServiceQueryThread.java
V 33
[root@uk-svn-2-new yoyodyne]#
Each time I've tries this the failure mode appears to be the same. The
initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the copied
over file, it starts with the 'id:' line directory. Is there anything I
can do to debug this further?
Tony.
Re: svnsync problems
Posted by Tony Sweeney <sw...@addr.com>.
On 02/20/15 10:21, Philip Martin wrote:
> You are making things very hard for yourself by running 1.6.11, it is
> very old and no longer maintained, unless your distribution is
> backporting fixes. A newer version will have fewer bugs and better
> performance.
You'll get no disagreement from me on that. At my last shop we made
sure to upgrade to the next major version any time the one we were
running went out of maintenance.
>
> 1.6 has a hotcopy bug
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=3596
>
> the effects are hard to predict but one possibility is corrupt revisions
> when committing to the hotcopy. You could delete the file
> db/rep-cache.db from the hotcopy and see if this fixes the problem. In
> 1.8 the hotcopy bug is fixed, 1.8 also attempts to detect and avoid
> problems caused by a corrupt rep-cache.db.
This was indeed the issue. We have a cron job that runs every five
minutes that wries a timestamp into a file and checks it in in order to
exercise our CI server. This is pretty much guaranteed to fire at least
once during the hotcopy, ensuring that the rep cache will be
inconsistent with the remainder of the hotcopy. Truncating the hotcopy
cache on the slave before starting the server allowed the svnsync to
proceed as expected.
Thank you very, very much for your help,
Tony.
>
> Tony Sweeney <sw...@addr.com> writes:
>
>> Hi,
>>
>> Mise en scene:
>>
>> We have three Subversion servers, version 1.6.11 running on
>> CentOS6. One is the live master and and the others are hot standbys in
>> different datacenters. The intent is that these are kept in sync with
>> the master using svnsync from a post-commit hook. One of the slaves
>> is already set up this way and has been since it was deployed by one
>> of my predecessors at this company. This is syncing nicely. The
>> other slave instance was also originally set up this way but was then
>> modified to instead use a block level filesystem mirroring technology,
>> again by one
>> of my predecessors. We have since concluded that the
>> mirroringsolution was not sufficiently trustworthy, and so we would
>> like to put it back to the svnsync method that it had run in the past.
>> The live master has in excess of 366,000 revisions and new revisions
>> come in at ~500 commits/day.
>>
>> One solution would be to simply create an empty repo, set the
>> necessary r0 revprops and hook scripts and sync from scratch.
>> However, we estimate that this would take 10-15 days due to the size
>> of the repository, so we would like to short-circuit this by starting
>> from a hotcopy. We take hotcopies nightly, save them to a backup
>> server and 'svnadmin verify' them there. I am trying to bring this
>> slave back on sync using the method described here:
>>
>> http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
>>
>> But using a hotcopy rather than a dump.
>>
>> The master hotcopy is restored in the correct location on the slave
>> and the hook scripts fixed up using Puppet (master and slave
>> necessarily have different hook script setups). Permissions are fixed
>> up to reflect that the hotcopy is taken as root but the server is
>> running under Apache. I set the svn:sync-last-merged-rev revprop on
>> r0 to reflect the newest revision on the hotcopy. At this point (or
>> so I thought) I should be able to replicate over the changes since
>> last night by running svnsync manually and then enable it in the hook
>> script.
>>
>> The problem is that that the revisions seem to be corrupted on
>> transfer to this slave.
>>
>> From the master:
>>
>> [root@uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive --trust-server-
>> cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
>> --sync-username svnsync --sync-password XXXXXXXX --source-username
>> svnsync --source-password XXXXXXXX --config-dir
>> /repos/svn/yoyodyne/hooks/certs
>> Transmitting file data ..
>> Committed revision 367219.
>> Copied properties for revision 367219.
>> Transmitting file data .
>> Committed revision 367220.
>> Copied properties for revision 367220.
>> Transmitting file data .
>> Committed revision 367221.
>> Copied properties for revision 367221.
>> Transmitting file data .svnsync: Corrupt representation '367220 0 41 29949930e5
>> 5f203ff7027917e765e6ca7d 2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
>> You have new mail in /var/spool/mail/root
>> [root@uk-svn-1-new yoyodyne]#
>>
>> In fact, checking on the slave, they have all come across corrupt:
>>
>> [root@uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
>> * Verified revision 367200.
>> * Verified revision 367201.
>> * Verified revision 367202.
>> * Verified revision 367203.
>> * Verified revision 367204.
>> * Verified revision 367205.
>> * Verified revision 367206.
>> * Verified revision 367207.
>> * Verified revision 367208.
>> * Verified revision 367209.
>> * Verified revision 367210.
>> * Verified revision 367211.
>> * Verified revision 367212.
>> * Verified revision 367213.
>> * Verified revision 367214.
>> * Verified revision 367215.
>> * Verified revision 367216.
>> * Verified revision 367217.
>> * Verified revision 367218.
>> svnadmin: Corrupt representation '367219 0 50
>> 633695f7c627c997b8b4065db2d4cb4db53c
>> 1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
>> svnadmin: Malformed representation header
>> You have mail in /var/spool/mail/root
>> [root@uk-svn-2-new svn]#
>>
>> Both the master and working slave verify all revisions clean at all
>> times. I've tried multiple hotcopies and every time the first
>> revision that it tries to sync comes across as corrupt.
>>
>> On the master the revision file looks as follows:
>>
>> [root@uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
>> DELTA 366873 14907 1619
>> SVN??@
>>
>> ?=??h?&ng serialVersionUID = 1;
>> ENDREP
>> DELTA 366873 16539 1063
>> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
>> requiredResults){
>>
>> hasMoreResults = false;
>> }
>> hasMoreResults;
>> }
>> }
>> ENDREP
>> id: 1j-366873.0-228442.r367219/295
>> type: file
>> pred: 1j-366873.0-228442.r366873/30632
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7wnq/_e
>> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/692
>> type: file
>> pred: 1h-366873.0-228442.r366873/32681
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7wnq/_d
>> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> [root@uk-svn-1-new yoyodyne]#
>>
>> The working slave looks as follows:
>>
>> [root@us-svn-1 yoyodyne]# head -30 db/revs/367/367219
>> DELTA 366873 14907 1619
>> SVN??@
>>
>> ?=??h?&ng serialVersionUID = 1;
>> ENDREP
>> DELTA 366873 16539 1063
>> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
>> requiredResults){
>>
>> hasMoreResults = false;
>> }
>> hasMoreResults;
>> }
>> }
>> ENDREP
>> id: 1j-366873.0-228442.r367219/295
>> type: file
>> pred: 1j-366873.0-228442.r366873/30631
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7yyk/_e
>> props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/692
>> type: file
>> pred: 1h-366873.0-228442.r366873/32680
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7yyk/_d
>> props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath: /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
>> edRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> [root@us-svn-1 yoyodyne]#
>>
>> But on the failing slave:
>> [root@uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
>> id: 1j-366873.0-228442.r367219/0
>> type: file
>> pred: 1j-366873.0-228442.r366873/30632
>> count: 1
>> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
>> 88900f096d5cd4913f 367218-7wnt/_e
>> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
>> copyroot: 228442 /MIME/MADb
>>
>> id: 1h-366873.0-228442.r367219/395
>> type: file
>> pred: 1h-366873.0-228442.r366873/32681
>> count: 1
>> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
>> b2df4c74a1710793 367218-7wnt/_d
>> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
>> cpath:
>> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
>> copyroot: 228442 /MIME/MADb
>>
>> PLAIN
>> K 19
>> CSVQueryThread.java
>> V 37
>> file 1e-366873.0-228442.r366873/31862
>> K 18
>> CachedRowList.java
>> V 35
>> file 1h-366873.0-228442.r367219/395
>> K 25
>> DBServiceQueryThread.java
>> V 33
>> [root@uk-svn-2-new yoyodyne]#
>>
>> Each time I've tries this the failure mode appears to be the same.
>> The initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the
>> copied over file, it starts with the 'id:' line directory. Is there
>> anything I can do to debug this further?
>>
>> Tony.
>>
>
Re: svnsync problems
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
> On Feb 20, 2015, at 8:21, Branko Čibej <br...@wandisco.com> wrote:
>
>> On 20.02.2015 13:59, Nico Kadel-Garcia wrote:
>> On Fri, Feb 20, 2015 at 5:50 AM, Philip Martin
>> <ph...@wandisco.com> wrote:
>>> "Andreas Stieger" <An...@gmx.de> writes:
>>>
>>>> In addition to what Philip wrote, for initializing a sync to a target
>>>> created via hotcopy, you will find "svnsync init --allow-non-empty"
>>>> very useful.
>>> But only after upgrading as that is not present in 1.6.
>>>
>>> --
>>> Philip Martin | Subversion Committer
>>> WANdisco // *Non-Stop Data*
>> CentOS 7 has subversion-1.7.14, and Wandisco publishes some very nice
>> binaries for RHEL and CentOS and Scientifici Linux that are up-to-date
>> and have some additional very nice features, like multip-master
>> servers.
>
> Correction: WANdisco does not publish Subversion binaries with a
> multi-master server. The multi-master replication is a separate product
> and not free at all.
>
> The Subversion binaries published by WANdisco are created from the
> stock, unpatched Subversion releases.
Thanks for the correction! I've tried to suggest that commercial solution to folks getting bogged down in backup and mirroring and high availability: it just solves *so many* issues.
Re: svnsync problems
Posted by Branko Čibej <br...@wandisco.com>.
On 20.02.2015 13:59, Nico Kadel-Garcia wrote:
> On Fri, Feb 20, 2015 at 5:50 AM, Philip Martin
> <ph...@wandisco.com> wrote:
>> "Andreas Stieger" <An...@gmx.de> writes:
>>
>>> In addition to what Philip wrote, for initializing a sync to a target
>>> created via hotcopy, you will find "svnsync init --allow-non-empty"
>>> very useful.
>> But only after upgrading as that is not present in 1.6.
>>
>> --
>> Philip Martin | Subversion Committer
>> WANdisco // *Non-Stop Data*
> CentOS 7 has subversion-1.7.14, and Wandisco publishes some very nice
> binaries for RHEL and CentOS and Scientifici Linux that are up-to-date
> and have some additional very nice features, like multip-master
> servers.
Correction: WANdisco does not publish Subversion binaries with a
multi-master server. The multi-master replication is a separate product
and not free at all.
The Subversion binaries published by WANdisco are created from the
stock, unpatched Subversion releases.
-- Brane
Re: svnsync problems
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Fri, Feb 20, 2015 at 5:50 AM, Philip Martin
<ph...@wandisco.com> wrote:
> "Andreas Stieger" <An...@gmx.de> writes:
>
>> In addition to what Philip wrote, for initializing a sync to a target
>> created via hotcopy, you will find "svnsync init --allow-non-empty"
>> very useful.
>
> But only after upgrading as that is not present in 1.6.
>
> --
> Philip Martin | Subversion Committer
> WANdisco // *Non-Stop Data*
CentOS 7 has subversion-1.7.14, and Wandisco publishes some very nice
binaries for RHEL and CentOS and Scientifici Linux that are up-to-date
and have some additional very nice features, like multip-master
servers.
If you want them, and I'm not saying these are production tested, my
personal tools for building more recent Subversion RPM's are at:
https://github.com/nkadel/subversion-1.8.x-srpm
https://github.ocm/nkadel/subversion-1.7.x-srpmj
I've been trying to get them into repoforge, but that seems to be moribund.
Re: svnsync problems
Posted by Philip Martin <ph...@wandisco.com>.
"Andreas Stieger" <An...@gmx.de> writes:
> In addition to what Philip wrote, for initializing a sync to a target
> created via hotcopy, you will find "svnsync init --allow-non-empty"
> very useful.
But only after upgrading as that is not present in 1.6.
--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*
Re: svnsync problems
Posted by Andreas Stieger <An...@gmx.de>.
> Tony Sweeney <sw...@addr.com> writes:
> > running under Apache. I set the svn:sync-last-merged-rev revprop on
> > r0 to reflect the newest revision on the hotcopy.
In addition to what Philip wrote, for initializing a sync to a target created via hotcopy, you will find "svnsync init --allow-non-empty" very useful.
Andreas
Re: svnsync problems
Posted by Philip Martin <ph...@wandisco.com>.
You are making things very hard for yourself by running 1.6.11, it is
very old and no longer maintained, unless your distribution is
backporting fixes. A newer version will have fewer bugs and better
performance.
1.6 has a hotcopy bug
http://subversion.tigris.org/issues/show_bug.cgi?id=3596
the effects are hard to predict but one possibility is corrupt revisions
when committing to the hotcopy. You could delete the file
db/rep-cache.db from the hotcopy and see if this fixes the problem. In
1.8 the hotcopy bug is fixed, 1.8 also attempts to detect and avoid
problems caused by a corrupt rep-cache.db.
Tony Sweeney <sw...@addr.com> writes:
> Hi,
>
> Mise en scene:
>
> We have three Subversion servers, version 1.6.11 running on
> CentOS6. One is the live master and and the others are hot standbys in
> different datacenters. The intent is that these are kept in sync with
> the master using svnsync from a post-commit hook. One of the slaves
> is already set up this way and has been since it was deployed by one
> of my predecessors at this company. This is syncing nicely. The
> other slave instance was also originally set up this way but was then
> modified to instead use a block level filesystem mirroring technology,
> again by one
> of my predecessors. We have since concluded that the
> mirroringsolution was not sufficiently trustworthy, and so we would
> like to put it back to the svnsync method that it had run in the past.
> The live master has in excess of 366,000 revisions and new revisions
> come in at ~500 commits/day.
>
> One solution would be to simply create an empty repo, set the
> necessary r0 revprops and hook scripts and sync from scratch.
> However, we estimate that this would take 10-15 days due to the size
> of the repository, so we would like to short-circuit this by starting
> from a hotcopy. We take hotcopies nightly, save them to a backup
> server and 'svnadmin verify' them there. I am trying to bring this
> slave back on sync using the method described here:
>
> http://www.cardinalpath.com/how-to-use-svnsync-to-create-a-mirror-backup-of-your-subversion-repository/
>
> But using a hotcopy rather than a dump.
>
> The master hotcopy is restored in the correct location on the slave
> and the hook scripts fixed up using Puppet (master and slave
> necessarily have different hook script setups). Permissions are fixed
> up to reflect that the hotcopy is taken as root but the server is
> running under Apache. I set the svn:sync-last-merged-rev revprop on
> r0 to reflect the newest revision on the hotcopy. At this point (or
> so I thought) I should be able to replicate over the changes since
> last night by running svnsync manually and then enable it in the hook
> script.
>
> The problem is that that the revisions seem to be corrupted on
> transfer to this slave.
>
> From the master:
>
> [root@uk-svn-1-new yoyodyne]# /usr/bin/svnsync --non-interactive --trust-server-
> cert sync https://uk-svn-2-new.uk.yoyodyne.lan/svn/yoyodyne
> --sync-username svnsync --sync-password XXXXXXXX --source-username
> svnsync --source-password XXXXXXXX --config-dir
> /repos/svn/yoyodyne/hooks/certs
> Transmitting file data ..
> Committed revision 367219.
> Copied properties for revision 367219.
> Transmitting file data .
> Committed revision 367220.
> Copied properties for revision 367220.
> Transmitting file data .
> Committed revision 367221.
> Copied properties for revision 367221.
> Transmitting file data .svnsync: Corrupt representation '367220 0 41 29949930e5
> 5f203ff7027917e765e6ca7d 2654a46e5fff1f98b747a77ebf600c10c895fbcc367219-7wnu/_6'
> You have new mail in /var/spool/mail/root
> [root@uk-svn-1-new yoyodyne]#
>
> In fact, checking on the slave, they have all come across corrupt:
>
> [root@uk-svn-2-new svn]# svnadmin verify -r 367200:HEAD /repos/svn/yoyodyne
> * Verified revision 367200.
> * Verified revision 367201.
> * Verified revision 367202.
> * Verified revision 367203.
> * Verified revision 367204.
> * Verified revision 367205.
> * Verified revision 367206.
> * Verified revision 367207.
> * Verified revision 367208.
> * Verified revision 367209.
> * Verified revision 367210.
> * Verified revision 367211.
> * Verified revision 367212.
> * Verified revision 367213.
> * Verified revision 367214.
> * Verified revision 367215.
> * Verified revision 367216.
> * Verified revision 367217.
> * Verified revision 367218.
> svnadmin: Corrupt representation '367219 0 50
> 633695f7c627c997b8b4065db2d4cb4db53c
> 1f15b1d8cea246d731676f43b2df4c74a1710793 367218-7wnt/_d'
> svnadmin: Malformed representation header
> You have mail in /var/spool/mail/root
> [root@uk-svn-2-new svn]#
>
> Both the master and working slave verify all revisions clean at all
> times. I've tried multiple hotcopies and every time the first
> revision that it tries to sync comes across as corrupt.
>
> On the master the revision file looks as follows:
>
> [root@uk-svn-1-new yoyodyne]# head -30 db/revs/367/367219
> DELTA 366873 14907 1619
> SVN??@
>
> ?=??h?&ng serialVersionUID = 1;
> ENDREP
> DELTA 366873 16539 1063
> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
> requiredResults){
>
> hasMoreResults = false;
> }
> hasMoreResults;
> }
> }
> ENDREP
> id: 1j-366873.0-228442.r367219/295
> type: file
> pred: 1j-366873.0-228442.r366873/30632
> count: 1
> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
> 88900f096d5cd4913f 367218-7wnq/_e
> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath:
> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
> copyroot: 228442 /MIME/MADb
>
> id: 1h-366873.0-228442.r367219/692
> type: file
> pred: 1h-366873.0-228442.r366873/32681
> count: 1
> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
> b2df4c74a1710793 367218-7wnq/_d
> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath:
> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
> copyroot: 228442 /MIME/MADb
>
> [root@uk-svn-1-new yoyodyne]#
>
> The working slave looks as follows:
>
> [root@us-svn-1 yoyodyne]# head -30 db/revs/367/367219
> DELTA 366873 14907 1619
> SVN??@
>
> ?=??h?&ng serialVersionUID = 1;
> ENDREP
> DELTA 366873 16539 1063
> SVN?J?7?V?)|???"?Y?? ??private boolean hasMoreResults = trueif(results.size() <
> requiredResults){
>
> hasMoreResults = false;
> }
> hasMoreResults;
> }
> }
> ENDREP
> id: 1j-366873.0-228442.r367219/295
> type: file
> pred: 1j-366873.0-228442.r366873/30631
> count: 1
> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
> 88900f096d5cd4913f 367218-7yyk/_e
> props: 366873 30579 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath:
> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
> copyroot: 228442 /MIME/MADb
>
> id: 1h-366873.0-228442.r367219/692
> type: file
> pred: 1h-366873.0-228442.r366873/32680
> count: 1
> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
> b2df4c74a1710793 367218-7yyk/_d
> props: 366873 32628 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath: /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/Cach
> edRowList.java
> copyroot: 228442 /MIME/MADb
>
> [root@us-svn-1 yoyodyne]#
>
> But on the failing slave:
> [root@uk-svn-2-new yoyodyne]# head -30 db/revs/367/367219
> id: 1j-366873.0-228442.r367219/0
> type: file
> pred: 1j-366873.0-228442.r366873/30632
> count: 1
> text: 367219 81 183 2999 2880cca7c6c56dea580d5c212595846c dc0cc36f718b468fc08817
> 88900f096d5cd4913f 367218-7wnt/_e
> props: 366873 30580 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath:
> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/DBServiceQueryThread.java
> copyroot: 228442 /MIME/MADb
>
> id: 1h-366873.0-228442.r367219/395
> type: file
> pred: 1h-366873.0-228442.r366873/32681
> count: 1
> text: 367219 0 50 6336 95f7c627c997b8b4065db2d4cb4db53c 1f15b1d8cea246d731676f43
> b2df4c74a1710793 367218-7wnt/_d
> props: 366873 32629 39 0 f2ea0bdf4310dae3aef66de358bcb0e1
> cpath:
> /MIME/MADb/trunk/server/src/main/java/com/yoyodyne/madb/query/memory/CachedRowList.java
> copyroot: 228442 /MIME/MADb
>
> PLAIN
> K 19
> CSVQueryThread.java
> V 37
> file 1e-366873.0-228442.r366873/31862
> K 18
> CachedRowList.java
> V 35
> file 1h-366873.0-228442.r367219/395
> K 25
> DBServiceQueryThread.java
> V 33
> [root@uk-svn-2-new yoyodyne]#
>
> Each time I've tries this the failure mode appears to be the same.
> The initial 'DELTA <E2><80><A6> ENDREP' sections are missing from the
> copied over file, it starts with the 'id:' line directory. Is there
> anything I can do to debug this further?
>
> Tony.
>
--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*