"Offset too large" error when packing repository in FSFS 7 format

[Radek Krotil <ra...@polarion.com>]

Hello.



Today, I encountered a problem when trying to pack a repository after
migrating it to the FSFS 7 format by performing full dump / load sequence.
Shortly, I get the following error
“Packing revisions in shard 5...svnadmin: E160056: Offset 391658998 too
large in revision 5102”

I was not able to understand from the documentation, what settings in
fsfs.conf should be modified to workaround this problem. Neither search in
the Internet brought any light into this. Is it even possible?



The mentioned revision seems to be a result of an user error. It resulted
in adding 169,200 paths to the repository. The delta file of the revision
5102 is 394141477 B. Subversion 1.9.3 was used when performing this
operation on CentOS 7.2. FSFS 7 backed uses default configuration
(therefore deltification is used).



Originally, the repository was stored in FSFS 6 format with default
settings (no deltification). Packing the repository in old format finished
successfully and the pack file size was about 831 MB.



Looking forward to hearing any guidance how to configure my backend so that
I can eventually pack the repository in the new format.



Thanks,

Radek Krotil

Re: "Offset too large" error when packing repository in FSFS 7 format

[Daniel Shahaf <d....@daniel.shahaf.name>]

Stefan Fuhrmann wrote on Wed, Jun 08, 2016 at 07:55:14 +0200:
> On 04.06.2016 18:57, Daniel Shahaf wrote:
> >Stefan Fuhrmann wrote on Sat, Jun 04, 2016 at 08:04:42 -0000:
> >>On 2016-06-03 09:36 (+0200), Radek Krotil <ra...@polarion.com> wrote:
> >>>Hello.
> >>>
> >>>Today, I encountered a problem when trying to pack a repository after
> >>>migrating it to the FSFS 7 format by performing full dump / load sequence.
> >>I assume you ran 'svnadmin load' onto a repository
> >>that was not accessible to the server at that time,
> >>so no remote user could accicentally write to it?
> >Why would that matter?  What could happen if somebody makes a commit or
> >a propedit in parallel to an 'svnadmin load'?  A concurrent commit will
> >cause mergeinfo in later revisions to have to have off-by-one errors,
> ( and copy-from info may break as well )
> >but shouldn't cause FS corruption.
> You are right in that it should not directly cause an issue.
> 
> What people tend to do, however, is to put the new repo
> at the same physical location as the old one and then the
> server might re-use outdated information from its cache.
> ATM, there is no known mechanism that will corrupt future
> commits in addition of just delivering bogus results while
> the cached data lasts.
> 

Still, the book does not warn about this problem: neither of
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.migrate
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.replication
has a warning about how replacing the repository at a given filesystem
path poisons the cache (by outdating it without invalidating it).

CC'ing svnbook-dev@.

> That said, it is dicey and since what we have here is looks
> like a new type of corruption (l2p translation appears to
> have worked but the result points a few bytes outside the
> valid range), I simply like to make sure ...

Fully agreed.

Thanks for clarifying,

Daniel

Re: "Offset too large" error when packing repository in FSFS 7 format

[Stefan Fuhrmann <st...@apache.org>]

On 04.06.2016 18:57, Daniel Shahaf wrote:
> Stefan Fuhrmann wrote on Sat, Jun 04, 2016 at 08:04:42 -0000:
>> On 2016-06-03 09:36 (+0200), Radek Krotil <ra...@polarion.com> wrote:
>>> Hello.
>>>
>>> Today, I encountered a problem when trying to pack a repository after
>>> migrating it to the FSFS 7 format by performing full dump / load sequence.
>> I assume you ran 'svnadmin load' onto a repository
>> that was not accessible to the server at that time,
>> so no remote user could accicentally write to it?
> Why would that matter?  What could happen if somebody makes a commit or
> a propedit in parallel to an 'svnadmin load'?  A concurrent commit will
> cause mergeinfo in later revisions to have to have off-by-one errors,
( and copy-from info may break as well )
> but shouldn't cause FS corruption.
You are right in that it should not directly cause an issue.

What people tend to do, however, is to put the new repo
at the same physical location as the old one and then the
server might re-use outdated information from its cache.
ATM, there is no known mechanism that will corrupt future
commits in addition of just delivering bogus results while
the cached data lasts.

That said, it is dicey and since what we have here is looks
like a new type of corruption (l2p translation appears to
have worked but the result points a few bytes outside the
valid range), I simply like to make sure ...
>>> Shortly, I get the following error
>>> “Packing revisions in shard 5...svnadmin: E160056: Offset 391658998 too
>>> large in revision 5102”
>> This is basically an "invalid access" error message.
>> Typical causes include repository corruption and
>> admins tinkering with the repository without informing
>> the server process. A maybe similar issue:
>>
>> https://issues.apache.org/jira/browse/SVN-4588
>>
>> In your case, however, the corruption is probably in
>> the repository itself. Please run 'svnadmin verify' on it.
>>
>>> I was not able to understand from the documentation, what settings in
>>> fsfs.conf should be modified to workaround this problem. Neither search in
>>> the Internet brought any light into this. Is it even possible?
>> This is most definitely not a configuration issue like
>> "your data is too large". Maybe, we should prefix the
>> error message with "invalid access" to prevent
>> confusion.
> How about being even more specific:
>
>      svnadmin: E1600NN: failed to locate representation of %s at revision %ld, offset %lld
>
> where %s identifies the origin of the offset value or the object that
> was expected to be located at that offset.
Well, we don't always have that much context information
available but we should definitely include the valid range
etc. People are then more likely to recognize this a problem
with the internal logic.

-- Stefan^2.

Re: "Offset too large" error when packing repository in FSFS 7 format

[Daniel Shahaf <d....@daniel.shahaf.name>]

Stefan Fuhrmann wrote on Sat, Jun 04, 2016 at 08:04:42 -0000:
> On 2016-06-03 09:36 (+0200), Radek Krotil <ra...@polarion.com> wrote: 
> > Hello.
> > 
> > Today, I encountered a problem when trying to pack a repository after
> > migrating it to the FSFS 7 format by performing full dump / load sequence.
> 
> I assume you ran 'svnadmin load' onto a repository
> that was not accessible to the server at that time,
> so no remote user could accicentally write to it?

Why would that matter?  What could happen if somebody makes a commit or
a propedit in parallel to an 'svnadmin load'?  A concurrent commit will
cause mergeinfo in later revisions to have to have off-by-one errors, 
but shouldn't cause FS corruption.

> > Shortly, I get the following error
> > “Packing revisions in shard 5...svnadmin: E160056: Offset 391658998 too
> > large in revision 5102”
> 
> This is basically an "invalid access" error message.
> Typical causes include repository corruption and
> admins tinkering with the repository without informing
> the server process. A maybe similar issue:
> 
> https://issues.apache.org/jira/browse/SVN-4588
> 
> In your case, however, the corruption is probably in
> the repository itself. Please run 'svnadmin verify' on it.
> 
> > I was not able to understand from the documentation, what settings in
> > fsfs.conf should be modified to workaround this problem. Neither search in
> > the Internet brought any light into this. Is it even possible?
> 
> This is most definitely not a configuration issue like
> "your data is too large". Maybe, we should prefix the
> error message with "invalid access" to prevent
> confusion.

How about being even more specific:

    svnadmin: E1600NN: failed to locate representation of %s at revision %ld, offset %lld

where %s identifies the origin of the offset value or the object that
was expected to be located at that offset.

?

Cheers,

Daniel

Re: "Offset too large" error when packing repository in FSFS 7 format

[Stefan Fuhrmann <st...@apache.org>]

On 2016-06-03 09:36 (+0200), Radek Krotil <ra...@polarion.com> wrote: 
> Hello.
> 
> Today, I encountered a problem when trying to pack a repository after
> migrating it to the FSFS 7 format by performing full dump / load sequence.

I assume you ran 'svnadmin load' onto a repository
that was not accessible to the server at that time,
so no remote user could accicentally write to it?

> Shortly, I get the following error
> \u201cPacking revisions in shard 5...svnadmin: E160056: Offset 391658998 too
> large in revision 5102\u201d

This is basically an "invalid access" error message.
Typical causes include repository corruption and
admins tinkering with the repository without informing
the server process. A maybe similar issue:

https://issues.apache.org/jira/browse/SVN-4588

In your case, however, the corruption is probably in
the repository itself. Please run 'svnadmin verify' on it.

> I was not able to understand from the documentation, what settings in
> fsfs.conf should be modified to workaround this problem. Neither search in
> the Internet brought any light into this. Is it even possible?

This is most definitely not a configuration issue like
"your data is too large". Maybe, we should prefix the
error message with "invalid access" to prevent
confusion. Subversion is likely to run out of memory
before hitting any repo format limitations.

> The mentioned revision seems to be a result of an user error. It resulted
> in adding 169,200 paths to the repository. The delta file of the revision
> 5102 is 394141477 B. Subversion 1.9.3 was used when performing this
> operation on CentOS 7.2. FSFS 7 backed uses default configuration
> (therefore deltification is used).

By "delta file" you mean the revision file (deb/revs/5/5102)?

If so, could you run a 'grep -oba L2P-INDEX 5102' on it
and post the results here?

How many revisions did the repo have before the
conversion?

> Originally, the repository was stored in FSFS 6 format with default
> settings (no deltification). Packing the repository in old format finished
> successfully and the pack file size was about 831 MB.

If the old repository still available, please re-run
the sump/load cycle to see whether the problem
is reproducible. Keep your current FSFS7 repo, too.

> Looking forward to hearing any guidance how to configure my backend so that
> I can eventually pack the repository in the new format.

My current best guess is that something interfered
with 'svnadmin load' process.

-- Stefan^2.