You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "nickdelnano (via GitHub)" <gi...@apache.org> on 2026/04/27 21:45:25 UTC

[PR] [Flink] Add option to avoid shuffle in bucket unaware append sink [paimon]

nickdelnano opened a new pull request, #7717:
URL: https://github.com/apache/paimon/pull/7717

   ### Purpose
   
   Bucket unaware append table [1] is a great choice for streaming events into Paimon format for batch consumers. These types of streams can be very high throughput like clickstream data. Currently there is a shuffle in the writer https://github.com/apache/paimon/pull/4203 and in my production use cases this shuffles a _lot_ of data.
   
   rebalance was added to avoid chaining, instead we may be able to use [startNewChain](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/overview/#task-chaining-and-resource-groups) to avoid the deadlock issue described in that PR without shuffle
   
   [1] https://cwiki.apache.org/confluence/display/PAIMON/PIP-6%3A+Unaware-Bucket+Table
   
   ### Tests
   
   FlinkCdcSyncTableSinkITCase.java has tests for schema evolution, however #4203 mentions this is hard to exercise in tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink] Add option to avoid shuffle in bucket unaware append sink [paimon]

Posted by "nickdelnano (via GitHub)" <gi...@apache.org>.
nickdelnano commented on PR #7717:
URL: https://github.com/apache/paimon/pull/7717#issuecomment-4338196074

   @JingsongLi I saw you authored https://github.com/apache/paimon/pull/4203 which relates to this PR, can you review it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink] Add option to avoid shuffle in bucket unaware append sink [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7717:
URL: https://github.com/apache/paimon/pull/7717#issuecomment-4396589480

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink] Add option to avoid shuffle in bucket unaware append sink [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7717:
URL: https://github.com/apache/paimon/pull/7717


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org