You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/09/17 12:11:35 UTC
[I] [server] Handle duplicate tiering completion reports idempotently [fluss]
luoyuxia opened a new issue, #4406:
URL: https://github.com/apache/fluss/issues/4406
### Search before asking
- [x] I searched the existing [issues](https://github.com/apache/fluss/issues).
Related: #3672 reports the same pair of state-transition errors during the first tiering round. This issue documents a reproducible duplicate-completion path and the resulting statistics overwrite; whether it also explains that report has not been confirmed.
### Fluss version
main (development). Reproduced against commit `746cdcb58ef04517241aa4b88de5e5ddfc1f158d` (`1.0-SNAPSHOT`).
### Please describe the bug 🐞
`LakeTableTieringManager.finishTableTiering()` processes a repeated normal completion report for the same table and tiering epoch after that round has already completed. It logs two invalid state transitions and overwrites the previous successful round's statistics.
Completion reports can overlap because the Flink tiering enumerator removes entries from `finishedTables` only after a heartbeat response returns. A second heartbeat can copy and send the same entry while the first request is still in flight. Repeated completion reports can also arise when a processed request's response is lost and the operation is retried, so the coordinator needs to handle duplicates safely.
**Reproduction**
1. Let table T enter `Pending`, then claim it with `requestTable()`. Its state is `Tiering` with epoch E.
2. Call `finishTableTiering(T, E, false, stats)`. The table transitions through `Tiered` to `Scheduled`.
3. Advance the clock slightly, keeping the table within its freshness interval.
4. Call `finishTableTiering(T, E, false, stats)` again before the next transition to `Pending`.
The second call produces:
```text
Fail to change state for table 42 from Scheduled to Tiered as it's not a valid state change.
Fail to change state for table 42 from Scheduled to Scheduled as it's not a valid state change.
```
An isolated reproduction compiled from the current enumerator and manager sources held the first completion response and invoked a reader split request before releasing it. Both requests carried the same table ID and epoch. With a manual clock, the results were:
| After processing | State | Last success time (ms) | Tiering duration (ms) |
| --- | --- | ---: | ---: |
| First completion | Scheduled | 1000 | 1000 |
| Duplicate completion | Scheduled | 1050 | -1 |
**Cause and impact**
In [`finishTableTiering()`](https://github.com/apache/fluss/blob/746cdcb58ef04517241aa4b88de5e5ddfc1f158d/fluss-server/src/main/java/org/apache/fluss/server/coordinator/LakeTableTieringManager.java#L444), epoch validation succeeds because the epoch is incremented on entering `Pending`, not on entering `Scheduled`. The method then updates tiering statistics before validating the state transition.
The first completion has already removed the start time, so the duplicate replaces the duration with `-1` and advances the last success timestamp. This also understates `tierLag`. The invalid transition handler logs and returns without throwing, so the caller proceeds to the second invalid transition and the heartbeat response reports no error for this completion.
The state remains `Scheduled` and the existing timer remains in place. This reproduction does not establish data loss or a stuck tiering job.
**Expected behavior**
A duplicate report for an already-completed normal tiering round should succeed without repeating state transitions, modifying the recorded result, or rescheduling the next round.
### Solution
Handle duplicate completion on the server, under the existing write lock:
- Preserve table-existence and epoch validation.
- Recognize a repeated completion of the already-completed normal round and return successfully before updating statistics or changing state.
- Validate completion eligibility before modifying the recorded result.
- Preserve stale-epoch fencing after the next round enters `Pending`, and preserve forced-completion behavior.
Add regression coverage for repeated completion in the same epoch, unchanged statistics and scheduling, and rejection of an old epoch after the next round starts. Client heartbeat scheduling changes can be handled separately.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [server] Handle duplicate tiering completion reports idempotently [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia closed issue #4406: [server] Handle duplicate tiering completion reports idempotently
URL: https://github.com/apache/fluss/issues/4406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [server] Handle duplicate tiering completion reports idempotently [fluss]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on issue #4406:
URL: https://github.com/apache/fluss/issues/4406#issuecomment-5724373165
> cc [@fhan688](https://github.com/fhan688)
Thanks! Please assign to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [server] Handle duplicate tiering completion reports idempotently [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #4406:
URL: https://github.com/apache/fluss/issues/4406#issuecomment-5714152782
cc @fhan688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org