You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Rishabh Daim (Jira)" <ji...@apache.org> on 2026/09/29 16:22:00 UTC

[jira] [Commented] (OAK-12408) Standalone cleanup() never journals gc.log leading to SNFE on cold standby / backup

    [ https://issues.apache.org/jira/browse/OAK-12408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120815#comment-18120815 ] 

Rishabh Daim commented on OAK-12408:
------------------------------------

lgtm, thanks [~shivamaggarwal] for the contribution.

> Standalone cleanup() never journals gc.log leading to SNFE on cold standby / backup
> -----------------------------------------------------------------------------------
>
>                 Key: OAK-12408
>                 URL: https://issues.apache.org/jira/browse/OAK-12408
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>    Affects Versions: 2.6.0
>            Reporter: Shivam
>            Assignee: Rishabh Daim
>            Priority: Critical
>             Fix For: 2.8.0
>
>
> {{AbstractGarbageCollectionStrategy.cleanup(Context)}} can run without a preceding local compaction (e.g. {{StandbyClientSync}} after sync, {{{}FileStoreBackupImpl{}}}/{{{}FileStoreBackupRestoreMBean{}}}). In that case it builds a placeholder via {{{}CompactionResult.skipped(...){}}}, whose {{requiresGCJournalEntry()}} is hard-coded to {{{}false{}}}:
> {code:java}
> static CompactionResult skipped(...) {
>     return new CompactionResult(currentGeneration, gcCount) {
>         // requiresGCJournalEntry() not overridden -> always false
>     };
> }
> {code}
> So {{GCJournal.persist(...)}} is never called for this path, even though the reclaimer *is* actively removing segments from older generations. On a cold standby, {{currentGeneration}} legitimately advances on every synced generation, but {{gc.log}} stays frozen and keeps pointing at a stale, already-superseded root.
> h3. Impact
>  * Standby's own {{gc.log}} never gains new entries after boot, even though sync + cleanup keep reclaiming segments from newer generations.
>  * After promoting the standby to primary, later compaction/cleanup cycles trust this stale {{gc.log}} entry, and reads of already-reclaimed segments throw {{{}SegmentNotFoundException{}}}.
> h3. Steps to reproduce
>  # Set up primary + cold standby.
>  # Run several rounds of content + online cleanup on primary so multiple generations/gc.log entries exist.
>  # Standby's {{segmentstore/gc.log}} shows no new entries despite "cleanup completed... space reclaimed" logs and on-disk generation clearly advancing.
>  # Promote standby to primary, keep running cleanup — eventually {{SegmentNotFoundException}} on segments already reclaimed while {{gc.log}} was frozen.
> h3. Suggested fix
> {code:java}
> @Override
> boolean requiresGCJournalEntry() {
>     return currentGeneration.isCompacted();
> }
> {code}
>  * {{GCJournal.persist}} already no-ops when the generation is unchanged, so this can't cause duplicate/bloated entries.
>  * Fresh, never-compacted repos are unaffected ({{{}isCompacted(){}}} is false at generation 0).
>  * {{{}succeeded(){}}}/{{{}aborted(){}}} paths untouched.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)