You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "wuchong (via GitHub)" <gi...@apache.org> on 2026/07/26 07:47:53 UTC

[I] [client] Sender handles a per-table RPC failure using destination-wide write batches [fluss]

wuchong opened a new issue, #3772:
URL: https://github.com/apache/fluss/issues/3772

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   ### Fluss version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `RecordAccumulator#drain` groups ready batches by destination TabletServer. The bucket list for one node is built across all physical tables, so a single drained list may contain batches from multiple tables:
   
   - [`RecordAccumulator#getAllBucketsInCurrentNode`](https://github.com/apache/fluss/blob/3a7c144b01b88812b1bc806e43897718f7384538/fluss-client/src/main/java/org/apache/fluss/client/write/RecordAccumulator.java#L898-L912)
   
   This is reachable in normal usage because all tables opened from one `FlussConnection` share the same `WriterClient`:
   
   - [`FlussConnection#getOrCreateWriterClient`](https://github.com/apache/fluss/blob/3a7c144b01b88812b1bc806e43897718f7384538/fluss-client/src/main/java/org/apache/fluss/client/FlussConnection.java#L119-L129)
   
   `Sender#sendWriteRequest` first creates one destination-wide `recordsByBucket`, then groups the same batches by table ID and sends one independent `ProduceLogRequest` or `PutKvRequest` per table. However, every RPC future captures the same destination-wide map:
   
   - [Build the node-wide map and split requests by table](https://github.com/apache/fluss/blob/3a7c144b01b88812b1bc806e43897718f7384538/fluss-client/src/main/java/org/apache/fluss/client/write/Sender.java#L371-L408)
   - [Pass the shared map to every RPC callback](https://github.com/apache/fluss/blob/3a7c144b01b88812b1bc806e43897718f7384538/fluss-client/src/main/java/org/apache/fluss/client/write/Sender.java#L425-L462)
   - [Apply an exceptional completion to every batch in that map](https://github.com/apache/fluss/blob/3a7c144b01b88812b1bc806e43897718f7384538/fluss-client/src/main/java/org/apache/fluss/client/write/Sender.java#L517-L529)
   
   As a result, an exceptional completion for table A is also applied to table B batches that belong to a different RPC:
   
   - A request-scoped non-retriable error (for example, table A is unauthorized while table B is authorized) can fail table B even if B's own request succeeds.
   - A request-scoped retriable error can re-enqueue unrelated table B batches.
   - If a connection failure completes several per-table RPC futures exceptionally, every callback iterates the full map. The same batch can therefore be re-enqueued multiple times. `RecordAccumulator#reEnqueue` does not deduplicate the batch before adding it to the deque, so this may duplicate sends for non-idempotent writers and corrupt attempts, retry metrics, or in-flight bookkeeping.
   
   A deterministic test can reproduce the problem as follows:
   
   1. Add ready batches for tables A and B whose bucket leaders are the same TabletServer.
   2. Drain them in one sender iteration, causing two table-specific RPCs.
   3. Complete A's future exceptionally while completing B's future successfully.
   4. Observe that B is failed or re-enqueued by A's callback.
   
   A second variant can complete both futures with a connection exception and verify that each batch is re-enqueued exactly once; currently each callback handles both batches.
   
   Expected behavior: each RPC callback should handle only the batches included in that RPC. The destination-wide map is only appropriate for the `gateway == null` path, where no table-specific RPC has been sent.
   
   ### Solution
   
   Build a per-table `Map<TableBucket, ReadyWriteBatch>` from each `writeBatches` list inside the `writeBatchByTable.forEach` loop and pass that map to the corresponding request/response callback. Keep the destination-wide handling for the `gateway == null` path.
   
   Add `SenderTest` coverage for:
   
   - two tables on the same node where only one per-table future fails;
   - a connection-level failure of both per-table futures, asserting that each batch is re-enqueued only once.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [client] Sender handles a per-table RPC failure using destination-wide write batches [fluss]

Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on issue #3772:
URL: https://github.com/apache/fluss/issues/3772#issuecomment-5089186719

   > cc [@luoyuxia](https://github.com/luoyuxia) [@platinumhamburg](https://github.com/platinumhamburg) [@swuferhong](https://github.com/swuferhong) , what do you think?
   
   This analysis is highly convincing. What we've observed so far is that whenever the cluster experiences jitter or abnormal conditions, client jobs often show extremely high retry load. I believe this issue has nailed the root cause, and it deserves to be fixed with high priority. I'd like to take a shot at fixing it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [client] Sender handles a per-table RPC failure using destination-wide write batches [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on issue #3772:
URL: https://github.com/apache/fluss/issues/3772#issuecomment-5083136256

   cc @luoyuxia @platinumhamburg @swuferhong , what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [client] Sender handles a per-table RPC failure using destination-wide write batches [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #3772:
URL: https://github.com/apache/fluss/issues/3772#issuecomment-5088553567

   Thanks for raising it. Both the analysis and the solution looks good to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [client] Sender handles a per-table RPC failure using destination-wide write batches [fluss]

Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg closed issue #3772: [client] Sender handles a per-table RPC failure using destination-wide write batches
URL: https://github.com/apache/fluss/issues/3772


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org