Re: 1.9.4 fix

[Johan Corveleyn <jc...@gmail.com>]

Op 27-apr.-2016 16:17 schreef "Terry Dooher" <Terry.Dooher@naturalmotion.com
>:
>
> Hi all,
>
>
>
> I’m in the process of dump/loading our repositories to take advantage of
FSFS format 7. Looking at the change list for 1.9.4 (out tomorrow), it
shows:
>
>        fsfs: fix a rare source of incomplete dump files and reports
(r1717876)
>
>
>
> I’m having trouble finding a layman’s explanation of what this issue
means, however. (I’ve read https://issues.apache.org/jira/browse/SVN-4554
but it doesn’t make much sense to me.) In practical terms, I’m running
through several hundred thousand revisions across 20 repositories. Is there
a chance this bug could happen silently and corrupt a dump in a way that
wouldn’t be picked up during svnadmin load (or by ‘svnadmin verify’
following the load)?
>
>
>
> I’m on 1.9.2 now. Should I wait for 1.9.4 to do this?
>

I would use at least 1.9.3 for the process running 'svnadmin dump', because
of this fix in 1.9.3:

* svnadmin dump: preserve no-op changes (r1709388 et al, issue #4598)

Otherwise your new repository might be slightly different from the original
regarding no-op changes (see the issue [1] for more explanation).

Concerning issue #4554 I'm not sure. From the issue description it sounds
like quite a rare edge case that you would only very rarely see in a real
world repository (and since it talks about dump files being broken, I'm
assuming you would detect it when loading the dump).

But maybe stefan2 or someone else knows more about the circumstances for
this issue, whether it's likely to occur in the wild, and what the visible
consequences would be?

-- 
Johan

[1] https://issues.apache.org/jira/plugins/servlet/mobile#issue/SVN-4598

Re: 1.9.4 fix

[Daniel Shahaf <d....@daniel.shahaf.name>]

Terry Dooher wrote on Tue, May 03, 2016 at 11:07:25 +0000:
> From: Stefan Fuhrmann [mailto:stefan2@apache.org] 29 April 2016 08:29
> > The best way to detect an incomplete dump is to run 'svn log -v -q'
> > on the repository root.  The output must be the same for the old and new repository.
> 
> That's good to know. Tests so far show all dumps/loads to succeed,
> with no errors in my case. I'll compare the logs to be certain,
> though.

Another option: dump the new repository and diff the dump to a dump of
the old repository.  This will catch far more problems than diffing 'log -v'
outputs.

There are a number of complications here:

- Deltas are compressed, so if you pass --deltas and the two producers
  used different zlib versions, the dumpfiles might differ even for
  the same repository instance.
  
- A dump without --deltas is quite large.

- There is no option to emit non-compressed deltas (svndiff0 rather than
  svndiff1).

Furthermore, there used to be a few sources of non-determinism (e.g.,
node header order and APR hash order), but I think Philip eliminated
them.  (I checked docs and history but couldn't determine which release
those changes first apepared in.)

So, in practice, I think one of these should work:

- cmp <(svnadmin1 dump --incremental old) <(svnadmin2 dump --incremental new)
  [ adding --deltas might introduce false positives ]

- cmp <(svnrdump2 dump url://old) <(svnadmin2 dump --deltas --incremental new)

- Run incremental, non-deltas, single-revision dumps on both
  repositories; checksum them; and compare the lists of (revnum, checksum).
  The lists should be identical.

All three variants could false positive if one of the producers
(generally the older one) was non-deterministic.

Cheers,

Daniel

P.S. <(${command}) is a shell construct (supported by bash and zsh,
possibly in the process of being standardised?) that evalutes to
the filename of a readable fifo producing the output of ${command};
e.g., �cat <(echo hello world)�.

RE: 1.9.4 fix

[Terry Dooher <Te...@naturalmotion.com>]

From: Stefan Fuhrmann [mailto:stefan2@apache.org] 29 April 2016 08:29

> On 28.04.2016 09:35, Johan Corveleyn wrote:
> > Op 27-apr.-2016 16:17 schreef "Terry Dooher"
> > <Terry.Dooher@naturalmotion.com
> > <ma...@naturalmotion.com>>:
> >  > I’m in the process of dump/loading our repositories to take
> > advantage of FSFS format 7. Looking at the change list for 1.9.4 (out tomorrow), it shows:
> >  >
> >  >        fsfs: fix a rare source of incomplete dump files and reports (r1717876)
> >  >
> >  > I’m having trouble finding a layman’s explanation of what this
> > issue means, however. (I’ve read
> > https://issues.apache.org/jira/browse/SVN-4554 but it doesn’t make
> > much sense to me.)
>
> Hi Terry,
>
> We recently fixed 2 issues that were caused by an idiosyncrasy of FSFS interacting badly with recent storage saving measures.
>
> Issue #4554 talks about two things, a set of conditions and their immediate impact (broken dump file).  The dump file will simply not load and that's easy enough to detect.  The set of conditions is as follows:
>
>    1. Representation sharing must be enabled (default since 1.6 or so).
>    2. At least one change must be committed using a 1.8+ server.
>    3. Property and directory deltification must not be enabled
>       (available since 1.8 but enabled by default only since 1.9).
>    4. A file gets committed whose contents happens to look exactly
>       like a serialized hash (i.e. a directory content or property list)
>       that has been committed with 1.8+.
>    5. No file with the same contents has been committed using a pre-1.8
>       server with rep-sharing enabled.
>
> Point 4 is unlikely to occur by accident.  By far your best shot at this is to have a 4-byte file containing "END\n", which happens to match an empty hash.  Another chance is committing the contents of a .svn folders of a pre-1.7 working copy - which requires some serious fiddling with the .svn folders.

That's a relief. Already this looks highly unlikely to occur in the wild, at least for us. No-one has (so far, fingers crossed) committed a .svn folder and we have had deltification enabled since we upgraded to 1.9 some time ago.

> These 5 conditions also apply to the issue fixed by r1717876 but the impact is different.  A change may be omitted entirely from the dump file, if further conditions are met:
>
>    6. The file was empty or thought to be empty (see 3. and 4.)
>       before the change and empty or thought to be empty (again
>       (3. and 4.) afterwards.
>    7. The dump is produced by a 1.9.0 - 1.9.3 server.
>
> E.g. if a file contains had "END\n" in r10 and changed to "" in r11, that change would not be dumped at all.  The dump file will not be broken but be incomplete.

> > Is there a chance this bug could happen silently and corrupt a dump in a way that
> > wouldn’t be picked up during svnadmin load (or by ‘svnadmin verify’ following the load)?
>
> 'svnadmin load' will fail.  The dump has no contents for the respective file node ('svnadmin dump' thought it was empty) but the MD5 and SHA1 stored in the dump don't match because they refer to the actual contents.
>
> The best way to detect an incomplete dump is to run 'svn log -v -q'
> on the repository root.  The output must be the same for the old and new repository.

That's good to know. Tests so far show all dumps/loads to succeed, with no errors in my case. I'll compare the logs to be certain, though.

> > I would use at least 1.9.3 for the process running 'svnadmin dump',
> > because of this fix in 1.9.3:
> > * svnadmin dump: preserve no-op changes (r1709388 et al, issue #4598)
> >
> That would be a third and unrelated issue.  It basically revolves around the question "Does SVN need to preserve a change attempt if it did not actually modify any contents?".  1.9.0 said "no", 1.9.3+ says "yes" again as would all older releases.
>
> No-op changes cannot be produced by standard SVN tooling but require direct low-level API access or some tool that writes its own dump files (e.g. as part of the conversion from CVS to SVN).  No contents was lost but the relation between commit message and changed paths list got lost.

We did an interim upgrade to 1.9.3 prior to starting this process, though it doesn't look like it will affect us as we don't have any API interactions with working copies beyond the usual clients.

> -- Stefan^2.
>

Thanks Stefan, that was a really comprehensive overview and it's certainly put my mind at ease.

Terry.

Re: 1.9.4 fix

[Daniel Shahaf <d....@daniel.shahaf.name>]

Stefan Fuhrmann wrote on Mon, May 02, 2016 at 07:19:34 +0200:
> On 30.04.2016 02:47, Daniel Shahaf wrote:
> >Does any third-party tool make that optimization?
> 
> Such a tool would probably ignore checksums altogether
> because there is no point in explicitly ignoring them
> for empty files only.

I thought a reader could, if a "Content-Length: 0" header was present,
hardcode the EXPECTED_MD5 value to d41d8cd98f00b204e9800998ecf8427e
instead of reading it from the Checksum-MD5 header.  It's not possible
to do this optimization for any non-zero Content-Length, and it still
provides the reader with assurance of transport integrity of the payload
itself.

What this optimization doesn't provide is assurance against certain bugs
in the dump producer, e.g., against a bug whereby the correct payload
and checksum, but incorrect Content-Length header, would be written,
*and* the payload would look like whatever the dumpstream parser
expects to follow an empty payload.

What's the worst-case failure mode of this optimization, then?
So far, I haven't come up with a failure mode that's both likely and
worse than a parse error on the next node.

Cheers,

Daniel
(by "payload" I mean the byte string whose md5 and length are in the
dump node headers)

Re: 1.9.4 fix

[Stefan Fuhrmann <st...@apache.org>]

On 30.04.2016 02:47, Daniel Shahaf wrote:
> Stefan Fuhrmann wrote on Fri, Apr 29, 2016 at 09:29:20 +0200:
>> Issue #4554 talks about two things, a set of conditions and their
>> immediate impact (broken dump file).  The dump file will simply
>> not load and that's easy enough to detect.  The set of conditions
>> is as follows:
>>
>>    1. Representation sharing must be enabled (default since 1.6 or so).
>>    2. At least one change must be committed using a 1.8+ server.
>>    3. Property and directory deltification must not be enabled
>>       (available since 1.8 but enabled by default only since 1.9).
>>    4. A file gets committed whose contents happens to look exactly
>>       like a serialized hash (i.e. a directory content or property list)
>>       that has been committed with 1.8+.
>>    5. No file with the same contents has been committed using a pre-1.8
>>       server with rep-sharing enabled.
>>
>> Point 4 is unlikely to occur by accident.  By far your best shot
>> at this is to have a 4-byte file containing "END\n", which happens
>> to match an empty hash.  Another chance is committing the contents
>> of a .svn folders of a pre-1.7 working copy - which requires some
>> serious fiddling with the .svn folders.
>
> If these conditions do occur in a repository, though, a third-party tool
> that does not verify checksums for empty files would incur data loss
> when accessing that repository through Subversion 1.9.3 or older.

There are two types of interfaces here:  Some SVN API
and the dump file.  Both are affected.  Invalid dump
files are easy to identify while API users would expect
file_length() to return correct results without double-
checking them, which it did not.

> Does any third-party tool make that optimization?

Such a tool would probably ignore checksums altogether
because there is no point in explicitly ignoring them
for empty files only.

>> 'svnadmin load' will fail.
>
> This sentence applies to any version of svnadmin, including pre-1.9.3
> ones.

Correct.

>> No-op changes cannot be produced by standard SVN tooling
>
> They can: just undo the changes while 'commit --editor-cmd' is waiting
> for the editor to exit.  (I posted a scripted version of that at some
> point, but can't find it.)

Thanks for pointing that out.  I was not aware that this
would send the change list before waiting for commit msg.

-- Stefan^2.

Re: 1.9.4 fix

[Daniel Shahaf <d....@daniel.shahaf.name>]

Stefan Fuhrmann wrote on Fri, Apr 29, 2016 at 09:29:20 +0200:
> Issue #4554 talks about two things, a set of conditions and their
> immediate impact (broken dump file).  The dump file will simply
> not load and that's easy enough to detect.  The set of conditions
> is as follows:
> 
>   1. Representation sharing must be enabled (default since 1.6 or so).
>   2. At least one change must be committed using a 1.8+ server.
>   3. Property and directory deltification must not be enabled
>      (available since 1.8 but enabled by default only since 1.9).
>   4. A file gets committed whose contents happens to look exactly
>      like a serialized hash (i.e. a directory content or property list)
>      that has been committed with 1.8+.
>   5. No file with the same contents has been committed using a pre-1.8
>      server with rep-sharing enabled.
> 
> Point 4 is unlikely to occur by accident.  By far your best shot
> at this is to have a 4-byte file containing "END\n", which happens
> to match an empty hash.  Another chance is committing the contents
> of a .svn folders of a pre-1.7 working copy - which requires some
> serious fiddling with the .svn folders.

If these conditions do occur in a repository, though, a third-party tool
that does not verify checksums for empty files would incur data loss
when accessing that repository through Subversion 1.9.3 or older.

Does any third-party tool make that optimization?

> 'svnadmin load' will fail.

This sentence applies to any version of svnadmin, including pre-1.9.3
ones.

> No-op changes cannot be produced by standard SVN tooling

They can: just undo the changes while 'commit --editor-cmd' is waiting
for the editor to exit.  (I posted a scripted version of that at some
point, but can't find it.)

Cheers,

Daniel

Re: 1.9.4 fix

[Stefan Fuhrmann <st...@apache.org>]

On 28.04.2016 09:35, Johan Corveleyn wrote:
> Op 27-apr.-2016 16:17 schreef "Terry Dooher" <Terry.Dooher@naturalmotion.com
> <ma...@naturalmotion.com>>:
>  >
>  > Hi all,
>  >
>  >
>  >
>  > I’m in the process of dump/loading our repositories to take advantage of FSFS
> format 7. Looking at the change list for 1.9.4 (out tomorrow), it shows:
>  >
>  >        fsfs: fix a rare source of incomplete dump files and reports (r1717876)
>  >
>  >
>  >
>  > I’m having trouble finding a layman’s explanation of what this issue means,
> however. (I’ve read https://issues.apache.org/jira/browse/SVN-4554 but it
> doesn’t make much sense to me.)

Hi Terry,

We recently fixed 2 issues that were caused by an idiosyncrasy
of FSFS interacting badly with recent storage saving measures.

Issue #4554 talks about two things, a set of conditions and their
immediate impact (broken dump file).  The dump file will simply
not load and that's easy enough to detect.  The set of conditions
is as follows:

   1. Representation sharing must be enabled (default since 1.6 or so).
   2. At least one change must be committed using a 1.8+ server.
   3. Property and directory deltification must not be enabled
      (available since 1.8 but enabled by default only since 1.9).
   4. A file gets committed whose contents happens to look exactly
      like a serialized hash (i.e. a directory content or property list)
      that has been committed with 1.8+.
   5. No file with the same contents has been committed using a pre-1.8
      server with rep-sharing enabled.

Point 4 is unlikely to occur by accident.  By far your best shot
at this is to have a 4-byte file containing "END\n", which happens
to match an empty hash.  Another chance is committing the contents
of a .svn folders of a pre-1.7 working copy - which requires some
serious fiddling with the .svn folders.

These 5 conditions also apply to the issue fixed by r1717876 but
the impact is different.  A change may be omitted entirely from
the dump file, if further conditions are met:

   6. The file was empty or thought to be empty (see 3. and 4.)
      before the change and empty or thought to be empty (again
      (3. and 4.) afterwards.
   7. The dump is produced by a 1.9.0 - 1.9.3 server.

E.g. if a file contains had "END\n" in r10 and changed to "" in
r11, that change would not be dumped at all.  The dump file will
not be broken but be incomplete.

  In practical terms, I’m running through several
> hundred thousand revisions across 20 repositories. Is there a chance this bug
> could happen silently and corrupt a dump in a way that wouldn’t be picked up
> during svnadmin load (or by ‘svnadmin verify’ following the load)?

'svnadmin load' will fail.  The dump has no contents for the
respective file node ('svnadmin dump' thought it was empty) but
the MD5 and SHA1 stored in the dump don't match because they refer
to the actual contents.

The best way to detect an incomplete dump is to run 'svn log -v -q'
on the repository root.  The output must be the same for the old
and new repository.

>  >
>  >
>  >
>  > I’m on 1.9.2 now. Should I wait for 1.9.4 to do this?
>  >
>
> I would use at least 1.9.3 for the process running 'svnadmin dump', because of
> this fix in 1.9.3:
>
> * svnadmin dump: preserve no-op changes (r1709388 et al, issue #4598)
>
> Otherwise your new repository might be slightly different from the original
> regarding no-op changes (see the issue [1] for more explanation).

That would be a third and unrelated issue.  It basically revolves
around the question "Does SVN need to preserve a change attempt
if it did not actually modify any contents?".  1.9.0 said "no",
1.9.3+ says "yes" again as would all older releases.

No-op changes cannot be produced by standard SVN tooling but require
direct low-level API access or some tool that writes its own dump
files (e.g. as part of the conversion from CVS to SVN).  No contents
was lost but the relation between commit message and changed paths
list got lost.

-- Stefan^2.

> [1] https://issues.apache.org/jira/plugins/servlet/mobile#issue/SVN-4598

RE: 1.9.4 fix

[Terry Dooher <Te...@naturalmotion.com>]

From: Johan Corveleyn [mailto:jcorvel@gmail.com]  Sent: 28 April 2016 08:36

> Op 27-apr.-2016 16:17 schreef "Terry Dooher" <Te...@naturalmotion.com>:
> >
> > I’m on 1.9.2 now. Should I wait for 1.9.4 to do this?
> >
> I would use at least 1.9.3 for the process running 'svnadmin dump', because of this fix in 1.9.3:
> * svnadmin dump: preserve no-op changes (r1709388 et al, issue #4598)
> Otherwise your new repository might be slightly different from the original regarding no-op changes (see the issue [1] for more explanation).

Thanks; that's a good point. I'll use a build of svnadmin 1.9.3 for the dumps/loads, at least; though I'll wait to see what Stefan has to say regarding #4554 before I go ahead.

Cheers,

Terry.