You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "diegomrsantos (via GitHub)" <gi...@apache.org> on 2026/09/11 08:33:09 UTC
[I] Purge can record completion after consumer offset directory sync fails [iggy]
diegomrsantos opened a new issue, #4128:
URL: https://github.com/apache/iggy/issues/4128
### Bug description
Purge can durably record its applied generation even when synchronizing a consumer offset directory fails. If a subsequent power loss loses the offset file deletions but preserves the purge generation, restart can restore the old consumer offsets without retrying the purge. Consumers can then skip messages written after the purge.
This affects both individual consumers and consumer groups. The failure path remains after #4092, including when consumer offset durability is `persisted`.
This report is based on code inspection. I have not injected the synchronization failure or reproduced the power loss sequence.
### Affected component and versions
Iggy server. Inspected `master` at `e0027506f2a0379ee3bb5508427ff570e3a9a435` and #4119 at `d89b81f0297cbcb6577a5a69cd44857c94b0cdf0`. The links below refer to the inspected `master` commit.
### Evidence
1. After deleting consumer offset files, `purge()` synchronizes both offset directories. A failure [only logs a warning](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7548-L7574). It does not return an error or preserve a pending cleanup obligation.
2. Execution can then [persist `purge.gen` and advance `applied_purge_generation`](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7627-L7654). Recording that generation synchronizes its own parent directory, which does not establish durability for deletions inside the separate consumer offset directories.
3. On restart, recovery [loads consumer and group offset files and seeds their values into partition state](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_helpers.rs#L181-L257). It can clamp a value to the recovered offset range, but does not discard it based on the purge generation. A clamped offset is still a consumer entry; "consumed through 0" is different from having no committed offset.
4. The reconciler [only requests another purge when the committed generation exceeds the applied generation](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_reconciler.rs#L1430-L1453). A surviving `purge.gen` therefore prevents the cleanup from being retried for that generation.
The durable journal marker added in #4092 does not close this gap. It is [written before offset cleanup](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7336-L7345). Its synchronization cannot cover deletions that occur later. Recovery [restores the journal purge floor](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L829-L832), but that floor does not invalidate the restored consumer offset files.
### Failure sequence derived from the code
1. Persist consumer and group offsets for an existing message history.
2. Purge the topic. Offset file deletions succeed, but synchronization of either offset directory fails.
3. Allow the subsequent write of `purge.gen` to succeed.
4. Lose power before the unsynchronized directory changes become durable. Model the old offset files surviving while the new purge generation survives too.
5. Restart. Recovery loads the old offsets, while the reconciler sees the purge generation as already applied.
6. Poll fresh messages with `Next`. Restored consumer progress can skip messages in the new history.
A process restart alone is not a reliable reproduction because the operating system may still retain and flush the pending directory changes.
### Expected behavior and suggested regression
The partition must not record the purge as locally applied while removal of its consumer offset files remains unconfirmed on durable storage. A synchronization failure must preserve a retry or recovery obligation. Recovery or retry must also avoid deleting fresh writes that the system has already acknowledged.
A deterministic regression should fail synchronization of each offset directory separately after successful deletion. It should verify that purge cannot reach an applied state that suppresses cleanup, then simulate recovery with the old offset files restored. After cleanup completes, both consumer offset maps should be empty and `Next` should return the complete fresh history. Cover both durability policies, a successful synchronization control, and preservation of any acknowledged fresh writes across cleanup retries.
### Related issues
- #3793 tracks old operations being replayed through journal repair after purge. This report concerns the durability of local file deletions and does not require repair from another replica.
- #4117, addressed by #4119, concerns stale poll completions changing progress after purge. This report does not require a pending poll.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Purge can record completion after consumer offset directory sync fails [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4128:
URL: https://github.com/apache/iggy/issues/4128#issuecomment-5795657198
thanks! all of it is covered by the plan in #4130 now, let's track it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Purge can record completion after consumer offset directory sync fails [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4128:
URL: https://github.com/apache/iggy/issues/4128#issuecomment-6061761900
closing, we're removing purge (see #4130). the removal pr goes up today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Purge can record completion after consumer offset directory sync fails [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed issue #4128: Purge can record completion after consumer offset directory sync fails
URL: https://github.com/apache/iggy/issues/4128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Purge can record completion after consumer offset directory sync fails [iggy]
Posted by "diegomrsantos (via GitHub)" <gi...@apache.org>.
diegomrsantos commented on issue #4128:
URL: https://github.com/apache/iggy/issues/4128#issuecomment-5751277422
@hubcio yeah, that makes sense. I added a smaller reproduction in #4130. It calls the real purge code and injects a sync failure for each offset directory. In both cases purge still succeeds and records the new generation. The control passes.
It doesn't simulate power loss or check `Next` after recovering the offsets, so your proposed test would cover what's missing.
I checked the retry part too. `purge_deferred` stops this replica from sending `PrepareOk`, but commits can still be applied before the next purge attempt. So we need to check that those fresh writes survive the retry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Purge can record completion after consumer offset directory sync fails [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4128:
URL: https://github.com/apache/iggy/issues/4128#issuecomment-5632296400
yes, `SimStorage` already supports directory-sync failures and `Crash::PowerLoss`, including restoring directory entries whose deletion wasn’t synced. that gives us the failure model needed here.
what needs plumbing is the production purge and recovery path. directory scans, offset-file deletion, syncing the consumer and consumer-group directories, and persisting/reloading `purge.gen` need to go through `DurableStorage`. recovery needs to reload offsets from the simulated filesystem, and `Next` checks must use that recovered state. the cluster simulator carries offsets across restart in memory, so its normal restart path won’t exercise this failure. the test should preserve the separate offset and marker directories, since syncing one doesn’t make changes in the others durable.
the existing storage purge test implements its own sequence and propagates directory-sync errors with `?`, while production logs them and continues. i’d reuse the production cleanup and completion code in that harness so the regression exercises the behavior we’re fixing.
then we can persist consumer and group offsets, successfully delete their files, inject a sync failure for each offset directory separately, allow subsequent operations, and trigger power loss. after recovery, we should verify that the completion marker can’t suppress required cleanup, stale offsets are gone once cleanup finishes, and `Next` returns the full fresh history. retries also need to preserve any acknowledged fresh writes. cover both durability policies and a successful-sync control.
@krishvishal does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org