[I] [Feature] Correct the out-of-order of data during one checkpoint [incubator-paimon]

["schnappi17 (via GitHub)" <gi...@apache.org>]

schnappi17 opened a new issue, #2778:
URL: https://github.com/apache/incubator-paimon/issues/2778

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   Paimon is already able to correct the disorder of data by specifying `sequence.field` to determine the order of updates. It's suitable for those cdc ingestion from business directly, which means that the business data update always triggers the update of `sequence.field`. 
   
   If your ingestion for Paimon table is based on the complex Flink cascade join, and the `sequence.field` comes from the left table. The join operator may generate many -U(-D), +U(+I) messages, which won't trigger the update for `sequence.field` at all, so there may be lots of -U(-D), +U(+I) messages with the same `sequence.field`, data disorder still can't be corrected even with the sequence auto padding. 
   
   Paimon can correct the disorder of data during a checkpoint when committing, like what `SinkUpsertMaterialize` node do, but do not rely on the state to do this.
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org