You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Julian Sedding <js...@apache.org> on 2018/12/03 13:16:03 UTC
Corrupt Oak segment store - how to recover last good revision?
Hi all
I'm analysing a segment store that was most likely corrupted after an
out-of-disk event. The system is running Oak 1.6.14.
Currently I am running oak-run check on the segment store, which is
unfortunately ~250GB in size after several system maintenance
operations being run last night (e.g. purgin of old versions, possibly
tar compaction, maybe others).
In case the check finds a "good" revision. I didn't find how to reset
it to that revision. Do I just remove all later journal entries?
Thanks & regards,
Julian
Re: Corrupt Oak segment store - how to recover last good revision?
Posted by Wim Symons <wi...@gmail.com>.
Just append it as the last line in journal.log. In case you were wrong, you still have the entire history.
Kind regards
Wim
> Op 3 dec. 2018 om 14:16 heeft Julian Sedding <js...@apache.org> het volgende geschreven:
>
> Hi all
>
> I'm analysing a segment store that was most likely corrupted after an
> out-of-disk event. The system is running Oak 1.6.14.
>
> Currently I am running oak-run check on the segment store, which is
> unfortunately ~250GB in size after several system maintenance
> operations being run last night (e.g. purgin of old versions, possibly
> tar compaction, maybe others).
>
> In case the check finds a "good" revision. I didn't find how to reset
> it to that revision. Do I just remove all later journal entries?
>
> Thanks & regards,
> Julian
Re: Fwd: Corrupt Oak segment store - how to recover last good revision?
Posted by Julian Sedding <js...@apache.org>.
Thanks! Will take a look. I was already quite happy with the existing
tooling, particularly I used the "oak-run check" and "oak-run history"
commands. Much faster than having to hack something myself :)
Regards
Julian
On Thu, Dec 6, 2018 at 10:52 AM Michael Dürig <md...@apache.org> wrote:
>
>
> Glad it worked!
>
> Have a look at the recent development around
> https://issues.apache.org/jira/browse/OAK-7866. Recovery should get much
> simpler and more powerful with this.
>
>
> On 06.12.18 10:39, Julian Sedding wrote:
> > This conversation seems to have dropped off the users list. Forwarding
> > to get it archived.
>
> My bad. My email client setup is a mess currently .
>
> Michael
>
> >
> > Regards
> > Julian
> >
> > ---------- Forwarded message ---------
> > From: Julian Sedding <js...@apache.org>
> > Date: Thu, Dec 6, 2018 at 10:38 AM
> > Subject: Re: Corrupt Oak segment store - how to recover last good revision?
> > To: Michael Dürig <md...@apache.org>
> > Cc: Julian Sedding <js...@apache.org>
> >
> >
> > Hi Michael
> >
> > Thanks for your reply!
> >
> > I was working on disaster recovery, particularly a segment tar store
> > (1.6.14) that got corrupted due to out-of-disk. Most likely during
> > online compaction. Therefore it was acceptable to lose any content
> > created after the corruption had occurred.
> >
> > Rather than append the revision to the journal, I truncated any later
> > revision from the journal. The net effect should be the same I expect.
> > In any case, it work fine and the repository is back in operation
> > since Monday night. After offline compaction of the segment store, its
> > size was reduced to ~25GB (still a lot IMHO but lots better than
> > 250GB).
> >
> > As things go, I used the same strategy again this week to recover a
> > tree of accidentally deleted nodes from another repository. So thanks
> > a lot for confirming the validity of this approach. Even though it
> > should not be used lightly and not without a good understanding of the
> > consequences.
> >
> > Regards
> > Julian
> > On Wed, Dec 5, 2018 at 8:31 PM Michael Dürig <md...@apache.org> wrote:
> >>
> >>
> >> Hi Julian
> >>
> >> Just append the found good revision to the end of the journal log
> >> following the format of the preceding lines. Afterwards you could run
> >> offline compaction to get rid of garbage. Please note though that this
> >> will roll back your repository to the identified good revision. All data
> >> from all later revisions will be lost.
> >>
> >> Michael
> >>
> >>
> >> On 03.12.18 14:16, Julian Sedding wrote:
> >>> Hi all
> >>>
> >>> I'm analysing a segment store that was most likely corrupted after an
> >>> out-of-disk event. The system is running Oak 1.6.14.
> >>>
> >>> Currently I am running oak-run check on the segment store, which is
> >>> unfortunately ~250GB in size after several system maintenance
> >>> operations being run last night (e.g. purgin of old versions, possibly
> >>> tar compaction, maybe others).
> >>>
> >>> In case the check finds a "good" revision. I didn't find how to reset
> >>> it to that revision. Do I just remove all later journal entries?
> >>>
> >>> Thanks & regards,
> >>> Julian
> >>>
Re: Fwd: Corrupt Oak segment store - how to recover last good
revision?
Posted by Michael Dürig <md...@apache.org>.
Glad it worked!
Have a look at the recent development around
https://issues.apache.org/jira/browse/OAK-7866. Recovery should get much
simpler and more powerful with this.
On 06.12.18 10:39, Julian Sedding wrote:
> This conversation seems to have dropped off the users list. Forwarding
> to get it archived.
My bad. My email client setup is a mess currently .
Michael
>
> Regards
> Julian
>
> ---------- Forwarded message ---------
> From: Julian Sedding <js...@apache.org>
> Date: Thu, Dec 6, 2018 at 10:38 AM
> Subject: Re: Corrupt Oak segment store - how to recover last good revision?
> To: Michael Dürig <md...@apache.org>
> Cc: Julian Sedding <js...@apache.org>
>
>
> Hi Michael
>
> Thanks for your reply!
>
> I was working on disaster recovery, particularly a segment tar store
> (1.6.14) that got corrupted due to out-of-disk. Most likely during
> online compaction. Therefore it was acceptable to lose any content
> created after the corruption had occurred.
>
> Rather than append the revision to the journal, I truncated any later
> revision from the journal. The net effect should be the same I expect.
> In any case, it work fine and the repository is back in operation
> since Monday night. After offline compaction of the segment store, its
> size was reduced to ~25GB (still a lot IMHO but lots better than
> 250GB).
>
> As things go, I used the same strategy again this week to recover a
> tree of accidentally deleted nodes from another repository. So thanks
> a lot for confirming the validity of this approach. Even though it
> should not be used lightly and not without a good understanding of the
> consequences.
>
> Regards
> Julian
> On Wed, Dec 5, 2018 at 8:31 PM Michael Dürig <md...@apache.org> wrote:
>>
>>
>> Hi Julian
>>
>> Just append the found good revision to the end of the journal log
>> following the format of the preceding lines. Afterwards you could run
>> offline compaction to get rid of garbage. Please note though that this
>> will roll back your repository to the identified good revision. All data
>> from all later revisions will be lost.
>>
>> Michael
>>
>>
>> On 03.12.18 14:16, Julian Sedding wrote:
>>> Hi all
>>>
>>> I'm analysing a segment store that was most likely corrupted after an
>>> out-of-disk event. The system is running Oak 1.6.14.
>>>
>>> Currently I am running oak-run check on the segment store, which is
>>> unfortunately ~250GB in size after several system maintenance
>>> operations being run last night (e.g. purgin of old versions, possibly
>>> tar compaction, maybe others).
>>>
>>> In case the check finds a "good" revision. I didn't find how to reset
>>> it to that revision. Do I just remove all later journal entries?
>>>
>>> Thanks & regards,
>>> Julian
>>>
Fwd: Corrupt Oak segment store - how to recover last good revision?
Posted by Julian Sedding <js...@apache.org>.
This conversation seems to have dropped off the users list. Forwarding
to get it archived.
Regards
Julian
---------- Forwarded message ---------
From: Julian Sedding <js...@apache.org>
Date: Thu, Dec 6, 2018 at 10:38 AM
Subject: Re: Corrupt Oak segment store - how to recover last good revision?
To: Michael Dürig <md...@apache.org>
Cc: Julian Sedding <js...@apache.org>
Hi Michael
Thanks for your reply!
I was working on disaster recovery, particularly a segment tar store
(1.6.14) that got corrupted due to out-of-disk. Most likely during
online compaction. Therefore it was acceptable to lose any content
created after the corruption had occurred.
Rather than append the revision to the journal, I truncated any later
revision from the journal. The net effect should be the same I expect.
In any case, it work fine and the repository is back in operation
since Monday night. After offline compaction of the segment store, its
size was reduced to ~25GB (still a lot IMHO but lots better than
250GB).
As things go, I used the same strategy again this week to recover a
tree of accidentally deleted nodes from another repository. So thanks
a lot for confirming the validity of this approach. Even though it
should not be used lightly and not without a good understanding of the
consequences.
Regards
Julian
On Wed, Dec 5, 2018 at 8:31 PM Michael Dürig <md...@apache.org> wrote:
>
>
> Hi Julian
>
> Just append the found good revision to the end of the journal log
> following the format of the preceding lines. Afterwards you could run
> offline compaction to get rid of garbage. Please note though that this
> will roll back your repository to the identified good revision. All data
> from all later revisions will be lost.
>
> Michael
>
>
> On 03.12.18 14:16, Julian Sedding wrote:
> > Hi all
> >
> > I'm analysing a segment store that was most likely corrupted after an
> > out-of-disk event. The system is running Oak 1.6.14.
> >
> > Currently I am running oak-run check on the segment store, which is
> > unfortunately ~250GB in size after several system maintenance
> > operations being run last night (e.g. purgin of old versions, possibly
> > tar compaction, maybe others).
> >
> > In case the check finds a "good" revision. I didn't find how to reset
> > it to that revision. Do I just remove all later journal entries?
> >
> > Thanks & regards,
> > Julian
> >