You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "lilei1128 (via GitHub)" <gi...@apache.org> on 2026/05/22 03:46:31 UTC
[PR] [server] Fix table deletion stuck when StopReplicaRequest send fails [fluss]
lilei1128 opened a new pull request, #3359:
URL: https://github.com/apache/fluss/pull/3359
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
- **Generative AI disclosure:** Indicate whether generative AI tools were used in authoring this PR. If yes, specify the tool below.
- [ ] No generative AI tools used
- [ ] Yes (please specify the tool below)
**(The sections below can be removed for hotfixes or typos)**
-->
<!--
Generated-by: [Tool Name and Version] following [the guidelines](https://github.com/apache/fluss/blob/main/AGENTS.md)
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
When a StopReplicaRequest send fails with a network-level throwable,
the coordinator had already transitioned replicas to ReplicaDeletionStarted
but silently dropped the error. This permanently stuck table deletion because:
- isEligibleForDeletion() blocks tables with any replica in ReplicaDeletionStarted
- resumeDeletions() only triggers from processDeleteReplicaResponseReceived,
which is only reached when a DeleteReplicaResponseReceivedEvent is emitted
- processDeadTabletServer() skips replicas that are toBeDeleted
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
Fix: when sendStopRequest() fails with a throwable, emit a
DeleteReplicaResponseReceivedEvent with error results for the
delete-flagged buckets (delete=true && deleteRemote=true). This feeds
into the existing retry/give-up logic in processDeleteReplicaResponseReceived,
which retries up to DELETE_TRY_TIMES (5) and then force-marks replicas as
ReplicaDeletionSuccessful, allowing deletion to complete.
Non-deletion stop replicas (delete=false) continue to silently ignore
send failures as before, since they are best-effort.
### Tests
<!-- List UT and IT cases to verify this change -->
Add testDeleteTableWithSendFailure() to verify that deletion completes
even when all StopReplicaRequest sends fail at the network level (i.e.,
CompletableFuture.failedFuture, not a response-level error code). The
existing testDeleteReplicaStateChange() only covered response-level errors.
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Fix table deletion stuck when StopReplicaRequest send fails [fluss]
Posted by "gyang94 (via GitHub)" <gi...@apache.org>.
gyang94 commented on PR #3359:
URL: https://github.com/apache/fluss/pull/3359#issuecomment-4554886565
Hi @lilei1128 , thanks for your contribution to fix this issue. Glad to see you are interested in this task.
In your PR, it is a good thinking to reuse the existing retry mechanism to solve this problem. However, the current retry loop doesn't truly solve the problem. The retry loop has no delay between attempts, so it degrades to "fail 5 times in milliseconds → pretend success → data residue on disk."
I'd like to solve this problem by introducing "**per-TS sender**" + "**ReplicaDeletionIneligible state**":
- Per-TS sender thread gives each TabletServer an independent request queue with backoff-aware retry. A temporarily unreachable TS doesn't burn through retry attempts instantly — the sender sleeps and retries over a meaningful time window.
- ReplicaDeletionIneligible state breaks the current all-or-nothing dilemma. Instead of force-marking unreachable replicas as successfully deleted, they enter Ineligible — which (1) unblocks isEligibleForDeletion (since Ineligible ≠ DeletionStarted, resumeDeletions is no longer stuck), and (2) preserves the truth that deletion hasn't happened yet. When the TS comes back online, replicas transition Ineligible → OfflineReplica → ReplicaDeletionStarted and get genuinely deleted.
If you are interested, welcome to review this [PR](https://github.com/apache/fluss/pull/3391), leave any comments there, or help to push code to fix based on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Fix table deletion stuck when StopReplicaRequest send fails [fluss]
Posted by "lilei1128 (via GitHub)" <gi...@apache.org>.
lilei1128 closed pull request #3359: [server] Fix table deletion stuck when StopReplicaRequest send fails
URL: https://github.com/apache/fluss/pull/3359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Fix table deletion stuck when StopReplicaRequest send fails [fluss]
Posted by "lilei1128 (via GitHub)" <gi...@apache.org>.
lilei1128 commented on PR #3359:
URL: https://github.com/apache/fluss/pull/3359#issuecomment-4554964831
> Hi @lilei1128 , thanks for your contribution to fix this issue. Glad to see you are interested in this task.
>
> In your PR, it is a good thinking to reuse the existing retry mechanism to solve this problem. However, the current retry loop doesn't truly solve the problem. The retry loop has no delay between attempts, so it degrades to "fail 5 times in milliseconds → pretend success → data residue on disk."
>
> I'd like to solve this problem by introducing "**per-TS sender**" + "**ReplicaDeletionIneligible state**":
>
> * Per-TS sender thread gives each TabletServer an independent request queue with backoff-aware retry. A temporarily unreachable TS doesn't burn through retry attempts instantly — the sender sleeps and retries over a meaningful time window.
> * ReplicaDeletionIneligible state breaks the current all-or-nothing dilemma. Instead of force-marking unreachable replicas as successfully deleted, they enter Ineligible — which (1) unblocks isEligibleForDeletion (since Ineligible ≠ DeletionStarted, resumeDeletions is no longer stuck), and (2) preserves the truth that deletion hasn't happened yet. When the TS comes back online, replicas transition Ineligible → OfflineReplica → ReplicaDeletionStarted and get genuinely deleted.
>
> If you are interested, welcome to review this [PR](https://github.com/apache/fluss/pull/3391), leave any comments there, or help to push code to fix based on it.
Hi @[reviewer_name],
Thank you so much for your detailed and insightful review! I really appreciate you taking the time to explain the flaws in the current retry approach and sharing the better design direction.
I fully agree with your points: the existing retry loop without delay can only cause rapid failures and leave data residues on disk, which doesn’t solve the root problem. The solution you proposed — combining **per-TS sender** with backoff-aware retry and **ReplicaDeletionIneligible** state — is elegant and thorough. It not only unblocks the deletion workflow but also keeps the correct state transition logic, which perfectly fixes the all-or-nothing dilemma.
I’m very interested in this design. I will take a close look at your PR.
Thanks again for your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Fix table deletion stuck when StopReplicaRequest send fails [fluss]
Posted by "lilei1128 (via GitHub)" <gi...@apache.org>.
lilei1128 commented on PR #3359:
URL: https://github.com/apache/fluss/pull/3359#issuecomment-4514824415
close #3357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org