You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "zhang-arvin (via GitHub)" <gi...@apache.org> on 2026/09/28 07:41:22 UTC
[PR] [#4489] [server] Re-arm KV snapshot scheduling after a transient initialization failure [fluss]
zhang-arvin opened a new pull request, #4492:
URL: https://github.com/apache/fluss/pull/4492
## Root cause
`PeriodicSnapshotManager.triggerSnapshot()` (fluss-server/src/main/java/org/apache/fluss/server/kv/snapshot/PeriodicSnapshotManager.java, lines 209-215):
```java
Optional<SnapshotRunnable> snapshotRunnableOptional;
try {
snapshotRunnableOptional = target.initSnapshot();
} catch (Exception e) {
LOG.error("Fail to init snapshot during triggering snapshot.", e);
return; // <-- scheduleNextSnapshot() never called
}
```
Scheduling in this class is a **one-shot chain**, not a repeating task: `start()` schedules
exactly one task, and every terminal branch of the trigger body is responsible for re-arming the
chain by calling `scheduleNextSnapshot()` — init-returns-empty (line 228), async success
(line 273) and async failure (line 285). The `catch` around `target.initSnapshot()` is the only
terminal branch that returns *without* re-arming.
So a single transient failure while the snapshot is being initialized (e.g. remote storage
unreachable during `initSnapshot()`) consumes the scheduled task and never submits a replacement.
The bucket's KV snapshot scheduler is dead until the TabletServer restarts, surfacing as exactly
one ERROR line followed by total silence, no retry and no recovery after the backend comes back —
as reproduced in #4489. The `numberOfConsecutiveFailures` counter was only maintained on the
async path (lines 251/278), so the init path also produced no failure count.
## Fix
Minimal change on the initialization-failure path: record the consecutive init failure, log it at
WARN together with the delay until the next attempt, and re-arm the chain via
`scheduleNextSnapshot()` — the same recovery semantics the existing async-phase failure branch
already uses. `numberOfConsecutiveFailures` is reset once initialization succeeds again.
A transient backend outage now degrades to missed snapshot cycles that recover automatically
instead of a permanently dead scheduler. No backoff was introduced on purpose, to keep both
failure paths in this class consistent.
## Tests
New `PeriodicSnapshotManagerTest#testScheduleNextSnapshotAfterInitFailure`: a `SnapshotTarget`
whose `initSnapshot()` throws on the first attempt and succeeds afterwards; it asserts that a next
snapshot is still scheduled after the failing attempt, and that the following attempt succeeds and
uploads the snapshot.
This test fails on the unpatched code at the re-arm assertion (no task is scheduled):
```
[ERROR] PeriodicSnapshotManagerTest.testScheduleNextSnapshotAfterInitFailure:162->checkOnlyOneScheduledTasks:214 » NoSuchElement
[ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0
```
With the fix applied:
```
mvn -pl fluss-server test -Dtest=PeriodicSnapshotManagerTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
```
`spotless:check` passes on the changed module.
Fixes #4489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#4489] [server] Re-arm KV snapshot scheduling after a transient initialization failure [fluss]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #4492:
URL: https://github.com/apache/fluss/pull/4492#issuecomment-6058856737
Thanks @midnattsol — the independent S3-outage verification is really helpful, and the shape you describe is exactly what #4489 was about. On the two offers:
**1. Consecutive-failures test — yes please, I'd take it.** The test currently in the PR only covers a single init failure followed by recovery. The long-outage shape (N init failures in a row, the chain re-armed on every attempt, the counter incrementing, then success clearing it) is the more valuable regression and directly matches the issue. If you open it against `zhang-arvin:fix/4489-kv-snapshot-init-failure-recovery` (or paste it here) I'll fold it into this PR with a `Co-authored-by` credit. One hook-up detail: the test target's `failNextInitSnapshot()` is currently a one-shot boolean, so for N consecutive failures it needs to become a count (e.g. `failNextInitSnapshots(n)`); the rest should slot in next to `testScheduleNextSnapshotAfterInitFailure`.
**2. Metric/gauge — agreed the observability half is worth closing, but I'd keep it out of this PR.** `numberOfConsecutiveFailures` is already incremented on both the sync init failure (new) and the async phase, but this class isn't wired to a `MetricGroup`, so exposing it as a gauge means plumbing it through the snapshot manager's construction. That's a clean follow-up rather than something to bolt on here — happy to take it after this lands, or a separate contribution is very welcome.
**3. Exponential-backoff variant — thanks for flagging.** I'm deliberately keeping fixed-interval semantics here so the retry matches the existing scheduling (`scheduleNextSnapshot()` uses `snapshotIntervalSupplier`); backoff would be a behavior change on its own and belongs in its own issue/discussion. No objection in principle.
Will add (1) as soon as it's up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#4489] [server] Re-arm KV snapshot scheduling after a transient initialization failure [fluss]
Posted by "midnattsol (via GitHub)" <gi...@apache.org>.
midnattsol commented on PR #4492:
URL: https://github.com/apache/fluss/pull/4492#issuecomment-5869219372
We fixed this independently in our fork and verified it live with real S3-outage drills (snapshots resume without restart), so I compared both approaches.
Two offers if useful:
1. A consecutive-failures test adapted to fixed-interval semantics (N init failures in a row, chain rearmed every time, success on recovery, counter incrementing). Your test covers a single failure; this covers the long-outage shape from the issue. Happy to contribute it here.
2. Your plan mentioned "plus a metric/log so the state is observable", but the PR only adds logs. The consecutive-failure count already exists in code. Exposing it (or the last-failure timestamp) as a gauge would close that half. I can contribute that too if you want it.
For the record, our fork carries an exponential-backoff variant (2× capped at 10×) for long outages; flagging in case it's of interest, no ask attached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org