You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "NOCERA, ANDY" <an...@att.com> on 2018/03/15 22:35:32 UTC
RE: E130003: The XML response contains invalid XML - Follow-up
Folks,
I used dump and load to debug the malformed node revision ID. Here are my steps and what learned. Looks like the revs' file text: entry has a zero instead of size. By just editing the size, verify worked. No other change was required. The question is can we correct this ourselves without a dump and load?
Thanks
Andy
# Notes from debugging E160062: Malformed node revision ID string after dump and load # rm -rf /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin create /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin dump -r0:3 wd2tva02 and /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin load /usr/tmp/xrepox
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.
/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 23.
/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify /usr/tmp/xrepox
* Verifying repository metadata ...
* Verified revision 2.
/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Error verifying revision 1.
svnadmin: E160062: Malformed node revision ID string
Compare dump and loaded repos revs 1-3
db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---
> text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
22c22
< _0.0.t0-4501db85ea8ce59fc8eed19225af3b1387e71cb10 add-dir false false /ccm06_prod_1_20387
---
> _0.0.t0-0 add-dir false false /ccm06_prod_1_20387
25c25
< 137 260
---
> 137 261
db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
diff db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
26c26
< text: 3 74 134 0 a714b2fcbeb1639111a020a133ae1028
---
> text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
30c30
< _0.0.t2-4501db85ea8ce59fc8eed19225af3b1387e71cb12 add-dir false false /ccm06_n1_1_21751
---
> _0.0.t2-2 add-dir false false /ccm06_n1_1_21751
33c33
< 221 346
---
> 221 348
db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
diff db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
22c22
< text: 2 76 92 0 83b62398481e74fcb4405d4739c25d5f
---
> text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
26c26
< _0.0.t1-4501db85ea8ce59fc8eed19225af3b1387e71cb11 add-dir false false /ccm06_prod_2_20387
---
> _0.0.t1-1 add-dir false false /ccm06_prod_2_20387
29c29
< 181 305
---
> 181 306
IF I CHANGE - wd2tva02/db/revs/0/1 to include 48 48 like the loaded repo verify passes
'text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917'
The Problem moves to the next rev, which I can also edit
$ /opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Error verifying revision 2.
svnadmin: E160062: Malformed node revision ID string
Looks like size field is zero for this repo grep text wd2tva02/db/revs/0/*
wd2tva02/db/revs/0/0:text: 0 0 4 4 2d2977d1c96f487abe4a1e202dd03b4e
wd2tva02/db/revs/0/1:text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
wd2tva02/db/revs/0/10:text: 10 77 432 0 65cdd97c3c194413c12ddc41f5fd7437
wd2tva02/db/revs/0/11:text: 11 76 476 0 29907530feba5dbcb9105e61f29f8a2d
wd2tva02/db/revs/0/12:text: 12 78 522 0 ccd453b94ffecc6bac34afc1e62b977f
wd2tva02/db/revs/0/13:text: 13 78 568 0 97965d3ec3d1d91bbf6f6bdf9c176570
wd2tva02/db/revs/0/14:text: 14 76 612 0 ff2281abf3bea67c63df9c83199b9288
wd2tva02/db/revs/0/15:text: 15 76 656 0 01f577ad55e77d72d9833a0e6a864093
wd2tva02/db/revs/0/16:text: 16 78 702 0 1a4a8868f7a819c3a5a0b049f158d2a3
wd2tva02/db/revs/0/17:text: 17 78 748 0 3b8c4d8e3c95415a5098c12750547c22
wd2tva02/db/revs/0/18:text: 18 76 792 0 88533d0f4d89abb917a54aa40c67379c
wd2tva02/db/revs/0/19:text: 19 76 836 0 90e8b5563d2f92ccf804b6b57d42a962
wd2tva02/db/revs/0/2:text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
wd2tva02/db/revs/0/20:text: 20 78 882 0 4730fe1d4df3ca5f4e9d2be2d6b48273
wd2tva02/db/revs/0/21:text: 21 78 928 0 5a39be128a193e69edd603e5464029c0
wd2tva02/db/revs/0/22:text: 22 75 971 0 23986c9522e8ce73fa93e7d6e4070d5b
wd2tva02/db/revs/0/23:text: 23 75 1014 0 6b6b656e3a76f227bf0806162da70859
wd2tva02/db/revs/0/3:text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
wd2tva02/db/revs/0/4:text: 4 74 176 176 aa41a08702fc9281d222519cb8a7b01e
wd2tva02/db/revs/0/5:text: 5 75 219 0 d134abf460f591ffc0c4d996b376fdd9
wd2tva02/db/revs/0/6:text: 6 75 262 0 cf9070ac92de06910dbb38427ef61535
wd2tva02/db/revs/0/7:text: 7 73 303 0 9b9d5dd52df70c31bd9930d826dce5f6
wd2tva02/db/revs/0/8:text: 8 73 344 0 ce7870a25dc9904d281a056820f3e118
wd2tva02/db/revs/0/9:text: 9 75 387 0 dec2dc2a8c2438c86a6a087d7fb03b82
After making several edits
/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Verified revision 2.
* Verified revision 3.
* Verified revision 4.
* Error verifying revision 5.
svnadmin: E160062: Malformed node revision ID string
#
# Notes from debugging E160062: Malformed node revision ID string after dump and load # rm -rf /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin create /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin dump -r0:3 wd2tva02 and /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin load /usr/tmp/xrepox
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.
/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 23.
/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify /usr/tmp/xrepox
* Verifying repository metadata ...
* Verified revision 2.
/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Error verifying revision 1.
svnadmin: E160062: Malformed node revision ID string
Compare dump and loaded repos revs 1-3
db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---
> text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
22c22
< _0.0.t0-4501db85ea8ce59fc8eed19225af3b1387e71cb10 add-dir false false /ccm06_prod_1_20387
---
> _0.0.t0-0 add-dir false false /ccm06_prod_1_20387
25c25
< 137 260
---
> 137 261
db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
diff db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
26c26
< text: 3 74 134 0 a714b2fcbeb1639111a020a133ae1028
---
> text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
30c30
< _0.0.t2-4501db85ea8ce59fc8eed19225af3b1387e71cb12 add-dir false false /ccm06_n1_1_21751
---
> _0.0.t2-2 add-dir false false /ccm06_n1_1_21751
33c33
< 221 346
---
> 221 348
db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
diff db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
22c22
< text: 2 76 92 0 83b62398481e74fcb4405d4739c25d5f
---
> text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
26c26
< _0.0.t1-4501db85ea8ce59fc8eed19225af3b1387e71cb11 add-dir false false /ccm06_prod_2_20387
---
> _0.0.t1-1 add-dir false false /ccm06_prod_2_20387
29c29
< 181 305
---
> 181 306
IF I CHANGE - wd2tva02/db/revs/0/1 to include 48 48 like the loaded repo verify passes
'text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917'
The Problem moves to the next rev, which I can also edit
$ /opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Error verifying revision 2.
svnadmin: E160062: Malformed node revision ID string
Looks like size field is zero for this repo grep text wd2tva02/db/revs/0/*
wd2tva02/db/revs/0/0:text: 0 0 4 4 2d2977d1c96f487abe4a1e202dd03b4e
wd2tva02/db/revs/0/1:text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
wd2tva02/db/revs/0/10:text: 10 77 432 0 65cdd97c3c194413c12ddc41f5fd7437
wd2tva02/db/revs/0/11:text: 11 76 476 0 29907530feba5dbcb9105e61f29f8a2d
wd2tva02/db/revs/0/12:text: 12 78 522 0 ccd453b94ffecc6bac34afc1e62b977f
wd2tva02/db/revs/0/13:text: 13 78 568 0 97965d3ec3d1d91bbf6f6bdf9c176570
wd2tva02/db/revs/0/14:text: 14 76 612 0 ff2281abf3bea67c63df9c83199b9288
wd2tva02/db/revs/0/15:text: 15 76 656 0 01f577ad55e77d72d9833a0e6a864093
wd2tva02/db/revs/0/16:text: 16 78 702 0 1a4a8868f7a819c3a5a0b049f158d2a3
wd2tva02/db/revs/0/17:text: 17 78 748 0 3b8c4d8e3c95415a5098c12750547c22
wd2tva02/db/revs/0/18:text: 18 76 792 0 88533d0f4d89abb917a54aa40c67379c
wd2tva02/db/revs/0/19:text: 19 76 836 0 90e8b5563d2f92ccf804b6b57d42a962
wd2tva02/db/revs/0/2:text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
wd2tva02/db/revs/0/20:text: 20 78 882 0 4730fe1d4df3ca5f4e9d2be2d6b48273
wd2tva02/db/revs/0/21:text: 21 78 928 0 5a39be128a193e69edd603e5464029c0
wd2tva02/db/revs/0/22:text: 22 75 971 0 23986c9522e8ce73fa93e7d6e4070d5b
wd2tva02/db/revs/0/23:text: 23 75 1014 0 6b6b656e3a76f227bf0806162da70859
wd2tva02/db/revs/0/3:text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
wd2tva02/db/revs/0/4:text: 4 74 176 176 aa41a08702fc9281d222519cb8a7b01e
wd2tva02/db/revs/0/5:text: 5 75 219 0 d134abf460f591ffc0c4d996b376fdd9
wd2tva02/db/revs/0/6:text: 6 75 262 0 cf9070ac92de06910dbb38427ef61535
wd2tva02/db/revs/0/7:text: 7 73 303 0 9b9d5dd52df70c31bd9930d826dce5f6
wd2tva02/db/revs/0/8:text: 8 73 344 0 ce7870a25dc9904d281a056820f3e118
wd2tva02/db/revs/0/9:text: 9 75 387 0 dec2dc2a8c2438c86a6a087d7fb03b82
After making several edits
/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Verified revision 2.
* Verified revision 3.
* Verified revision 4.
* Error verifying revision 5.
svnadmin: E160062: Malformed node revision ID string
Can we edit the revs 'text:" entry instead of a dump a load?
-----Original Message-----
From: Johan Corveleyn [mailto:jcorvel@gmail.com]
Sent: Wednesday, March 07, 2018 6:27 AM
To: NOCERA, ANDY <an...@att.com>
Cc: users@subversion.apache.org
Subject: Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http
On Tue, Mar 6, 2018 at 5:21 PM, NOCERA, ANDY <an...@att.com> wrote:
> John,
>
> Good feedback.
>
> Svnadmin 1.9 verify is showing the error also. I did a dump and load and it resolved that repo. Having reviewed my repos, it looks like around 30% have this issue. Not sure what we could have done wrong to cause it. A real simple repo is mine and has only a few commits shows the error. Is there a correction method quicker than a dump and load?
>
Nice, great that dump+load fixes it. I don't think there is a quicker method.
It might be worth investigating why this happened to begin with. But I don't really know where to start. One hypothesis is that this corruption is already lingering there for years (until 1.9 it wouldn't have been noticed) ... perhaps something outside of SVN manipulated the rev files years ago? Or perhaps there was a bug once in SVN that caused this to happen (but the corruption remained benign / unnoticed, until the stricter validation by 1.9). It's also possible that the stricter validation by 1.9 contains a bug, and is too strict for some cases (though in that case I would have expected more reports on this list).
Maybe you can make a more accurate hypothesis by investigating exactly what the "Malformed node revision ID string"s looks like. Actually, danielsh just improved that error message a few days ago, by adding the actual data to the error message:
https://urldefense.proofpoint.com/v2/url?u=http-3A__svn.apache.org_viewvc-3Fview-3Drevision-26revision-3D1825846&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lHImfy5Z3mUV3q_wXtPXpg&m=H1vU79y-LZUNg2o-MTOUnApkEdSPmq6uvVWkpF1n-ZM&s=VcsP7Qwl3TAy_zSkzRZaG5vEBgC4alj__qhGeBQr_54&e=
So if you can build svn from source you might be able to perform a build from the latest svn trunk, and run 'svnadmin verify' to get the more verbose error message (be careful not to use your trunk svn build on production data without creating a backup of course). Or alternatively you can try taking a look into the rev files yourself, to find the "malformed node revision ID".
--
Johan
fsfs 'structure' terminology: length/size v. size/expanded-size (was:
Re: E130003: The XML response contains invalid XML - Follow-up)
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
[ cc += dev@ ]
NOCERA, ANDY wrote on Thu, 15 Mar 2018 22:35 +0000:
> Folks,
>
> I used dump and load to debug the malformed node revision ID. Here are
> my steps and what learned. Looks like the revs' file text: entry has
> a zero instead of size.
To be clear, you mean the fourth field. On line 562 'size' is the name
of the fourth field but on line 119, and in the representation_t struct,
'size' is the name of the third field.[1]
This is a recipe for confusion. Normally I think of the fields using
the size/expanded-size terminology so I propose to change 'structure' to
use that terminology in lieu of length/size (and leave a note behind
pointing out that old versions of the document used <size> differently).
WDYT?
Daniel
[1] All line numbers refer to subversion/libsvn_fs_fs/structure.
> diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> 18c18
> < text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
> ---
> > text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
> 22c22
Re: E130003: The XML response contains invalid XML - Follow-up
Posted by Johan Corveleyn <jc...@gmail.com>.
On Fri, Mar 16, 2018 at 4:57 PM, Philip Martin <ph...@codematters.co.uk> wrote:
> Daniel Shahaf <d....@daniel.shahaf.name> writes:
>
>> Philip Martin wrote on Fri, 16 Mar 2018 13:44 +0000:
>>
>> Changing "0" to "48" would also have broken the <root-offset> and
>> <cp-offset> offsets in that revision file, so how come 'verify' worked after
>> that change?
>
> In the examples he gave it looks like the root node itself is being
> edited. That works because the change is after <root-offset> but before
> <cp-offset> and he shows <cp-offset> changing as well.
>
> I guess you can get away with editing the last node in the file provided
> you also change <cp-offset> in the same file.
Does that also explain why the OP could repair some repositories by
simply dumping and loading them? Or would dump+load also work for
corruption-instances that are not on the root node?
From the perspective of the recoveries done by the OP, this doesn't
seem like a "breaking corruption", since it can be recovered from
quite easily.
If that is not the case, and some unrecoverable instances remain (that
cannot be dumped+loaded), can we offer any other suggestions to
recover?
--
Johan
Re: E130003: The XML response contains invalid XML - Follow-up
Posted by Philip Martin <ph...@codematters.co.uk>.
Daniel Shahaf <d....@daniel.shahaf.name> writes:
> Philip Martin wrote on Fri, 16 Mar 2018 13:44 +0000:
>
> Changing "0" to "48" would also have broken the <root-offset> and
> <cp-offset> offsets in that revision file, so how come 'verify' worked after
> that change?
In the examples he gave it looks like the root node itself is being
edited. That works because the change is after <root-offset> but before
<cp-offset> and he shows <cp-offset> changing as well.
I guess you can get away with editing the last node in the file provided
you also change <cp-offset> in the same file.
--
Philip
Re: E130003: The XML response contains invalid XML - Follow-up
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Philip Martin wrote on Fri, 16 Mar 2018 13:44 +0000:
> "NOCERA, ANDY" <an...@att.com> writes:
>
> > I used dump and load to debug the malformed node revision ID. Here
> > are my steps and what learned. Looks like the revs' file text: entry
> > has a zero instead of size. By just editing the size, verify worked.
> > No other change was required. The question is can we correct this
> > ourselves without a dump and load?
> >
> > db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> > diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> > 18c18
> > < text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
> > ---
> > > text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
> > 22c22
>
> That looks like issue 4554
>
> https://issues.apache.org/jira/projects/SVN/issues/SVN-4554
>
> Editing the file is unlikely to work. Later revisons refer to data in
> earlier revisions via a byte offset into the earlier file. When you
> edit "0" to "48" you have changed the byte offset of all the data beyond
> the edit and that breaks the references in all the later files.
Changing "0" to "48" would also have broken the <root-offset> and
<cp-offset> offsets in that revision file, so how come 'verify' worked after
that change?
Cheers,
Daniel
(The <foo-offset> terms are from the 'structure' file and describe
format 6; format 7 has <l2p offset> and <p2l offset> which are
analogous)
Re: E130003: The XML response contains invalid XML - Follow-up
Posted by Philip Martin <ph...@codematters.co.uk>.
"NOCERA, ANDY" <an...@att.com> writes:
> I used dump and load to debug the malformed node revision ID. Here
> are my steps and what learned. Looks like the revs' file text: entry
> has a zero instead of size. By just editing the size, verify worked.
> No other change was required. The question is can we correct this
> ourselves without a dump and load?
>
> db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> 18c18
> < text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
> ---
> > text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
> 22c22
That looks like issue 4554
https://issues.apache.org/jira/projects/SVN/issues/SVN-4554
Editing the file is unlikely to work. Later revisons refer to data in
earlier revisions via a byte offset into the earlier file. When you
edit "0" to "48" you have changed the byte offset of all the data beyond
the edit and that breaks the references in all the later files. You
would need to identify all the affected offsets in later files and
modify them, which in turn may lead to more offset changes. You may
need to recompute some checksums as well.
--
Philip
fsfs 'structure' terminology: length/size v. size/expanded-size (was:
Re: E130003: The XML response contains invalid XML - Follow-up)
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
[ cc += dev@ ]
NOCERA, ANDY wrote on Thu, 15 Mar 2018 22:35 +0000:
> Folks,
>
> I used dump and load to debug the malformed node revision ID. Here are
> my steps and what learned. Looks like the revs' file text: entry has
> a zero instead of size.
To be clear, you mean the fourth field. On line 562 'size' is the name
of the fourth field but on line 119, and in the representation_t struct,
'size' is the name of the third field.[1]
This is a recipe for confusion. Normally I think of the fields using
the size/expanded-size terminology so I propose to change 'structure' to
use that terminology in lieu of length/size (and leave a note behind
pointing out that old versions of the document used <size> differently).
WDYT?
Daniel
[1] All line numbers refer to subversion/libsvn_fs_fs/structure.
> diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
> 18c18
> < text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
> ---
> > text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
> 22c22