You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "fhan688 (via GitHub)" <gi...@apache.org> on 2026/09/20 11:24:45 UTC

[PR] [server] Handle duplicate tiering completion reports idempotently [fluss]

fhan688 opened a new pull request, #4445:
URL: https://github.com/apache/fluss/pull/4445

   ### Purpose
   
     Linked issue: close #4406
   
     Tiering completion reports are delivered with at-least-once semantics and may overlap while an earlier heartbeat is still in flight. A repeated normal completion for the same table and tiering epoch currently passes epoch validation, overwrites the previously recorded tiering result, and then attempts two invalid state transitions.
   
     This change makes duplicate normal completion reports idempotent and ensures that completion eligibility is validated before tiering statistics are modified.
   
   ### Brief change log
   
     - Treat a repeated normal completion for an already completed, non-zero tiering epoch as a successful no-op.
     - Preserve table-existence and tiering-epoch validation before duplicate detection.
     - Require non-duplicate completion reports to be in the `Tiering` state before updating tiering statistics.
     - Preserve stale-epoch fencing after the next tiering round starts.
     - Preserve the existing forced-completion behavior, including immediate transition to the next epoch.
     - Add regression coverage for:
       - duplicate normal completion reports;
       - unchanged completion timestamp, duration, file size, and record count;
       - unchanged scheduling of the next tiering round;
       - rejection of stale completion reports after the next round starts;
       - rejection of completion reports from ineligible states;
       - fencing of repeated forced-completion reports.
   
   ### Tests
   
     - `LakeTableTieringManagerTest`
       - 11 tests passed with no failures or errors.
     - Spotless check passed.
     - Checkstyle passed.
     - Apache RAT check passed.
   
     Test command:
   
     ```bash
     mvn -pl fluss-server -am \
       -DskipITs \
       -Dcheckstyle.skip=true \
       -DfailIfNoTests=false \
       -Dsurefire.failIfNoSpecifiedTests=false \
       -Dtest=LakeTableTieringManagerTest \
       test
   ```
   
   ### API and Format
   
     No public API, RPC protocol, or storage format changes.
   
   ### Documentation
   
     No documentation changes are required because this is an internal server-side correctness fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [server] Handle duplicate tiering completion reports idempotently [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #4445:
URL: https://github.com/apache/fluss/pull/4445


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [server] Handle duplicate tiering completion reports idempotently [fluss]

Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on PR #4445:
URL: https://github.com/apache/fluss/pull/4445#issuecomment-5756520473

   Thanks for the review, @luoyuxia. I’ve addressed all the comments:
   
     - Simplified duplicate detection to treat the `Scheduled` state as an idempotent no-op, with a debug log and an explanation of the at-least-once delivery behavior.
     - Removed the additional `Tiering` state validation to keep this PR scoped to duplicate completion handling.
     - Folded the duplicate completion assertions into the existing scheduling lifecycle test and removed the redundant standalone tests.
     - Kept the repeated forced-completion fencing assertion in the existing force-finish test.
   
     `LakeTableTieringManagerTest` passes with all 9 tests, and Spotless, Checkstyle, and RAT checks also pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [server] Handle duplicate tiering completion reports idempotently [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4445:
URL: https://github.com/apache/fluss/pull/4445#discussion_r4059175257


##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/LakeTableTieringManager.java:
##########
@@ -447,6 +447,20 @@ public void finishTableTiering(
                 lock,
                 () -> {
                     validateTieringServiceRequest(tableId, tieredEpoch);
+                    TieringState tieringState = tieringStates.get(tableId);
+                    if (!isForceFinished

Review Comment:
   ```suggestion
                     if (tieringState == TieringState.Scheduled) {
         // Completion reports are delivered at least once. Overlapping heartbeats may copy the
         // same finishedTables entry before the first response removes it, and a lost response
         // may also cause the report to be retried. The first report has already moved the table
         // to Scheduled, so ignore subsequent reports to preserve the recorded statistics and
         // avoid invalid state transitions.
         LOG.debug(
                 "Ignore the duplicate tiering completion report for table {} at epoch {} "
                         + "because the table is already in Scheduled state.",
                 tableId,
                 tieredEpoch);
         return;
     }
   ```



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/LakeTableTieringManager.java:
##########
@@ -447,6 +447,20 @@ public void finishTableTiering(
                 lock,
                 () -> {
                     validateTieringServiceRequest(tableId, tieredEpoch);
+                    TieringState tieringState = tieringStates.get(tableId);
+                    if (!isForceFinished
+                            && tieredEpoch > 0
+                            && tieringState == TieringState.Scheduled) {
+                        // The normal tiering round has already completed. Treat a repeated report
+                        // for the same epoch as a successful no-op.
+                        return;
+                    }
+                    if (tieringState != TieringState.Tiering) {

Review Comment:
   I think we can remove this 



##########
fluss-server/src/test/java/org/apache/fluss/server/coordinator/LakeTableTieringManagerTest.java:
##########
@@ -175,6 +175,91 @@ void testFinishTableTieringReTriggerSchedule() {
         assertRequestTable(tableId1, tablePath1, 2);
     }
 
+    @Test
+    void testDuplicateNormalFinishIsIdempotent() {

Review Comment:
   Could we simplify the regression coverage by extending the existing `testFinishTableTieringReTriggerSchedule` above instead of adding a separate full-lifecycle test? After the first successful finish, send the same completion again with different stats, then assert that the completion timestamp, duration, file size, record count, and original scheduling remain unchanged. The existing test can still advance to the next round and verify epoch 2.
   
   With the proposed `Scheduled` no-op semantics, the initial-Scheduled half of `testFinishTableTieringRequiresTieringState` is no longer applicable; that test can be removed or reduced to only the remaining state behavior. Stale-epoch fencing is already covered by `testTieringServiceTimeOutReTriggerPending`, while the short repeated force-finish assertion can stay in the force-finish test. This should keep all important behavior covered while reducing the added test code substantially.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org