[jira] [Created] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

Tzu-Li (Gordon) Tai created FLINK-6349:
------------------------------------------

             Summary: Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
                 Key: FLINK-6349
                 URL: https://issues.apache.org/jira/browse/FLINK-6349
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


As described in FLINK-6316, currently the Kinesis consumer does not provide any ordering guarantees when resharding occurs.

While this cannot be enforced globally (i.e. if a merged / split shard's child shard ends up in a different subtask, we cannot do any coordination for ordering guarantee), we can definitely enforce this locally for each subtask. Simply put, we can still locally enforce ordering by making sure that discovered child shards are consumed only after any of its parent shards that were on the same subtask are fully consumed.

To do this, we would also need to add "parent shard" information to {{KinesisStreamShard}} (Flink's representation of Kinesis shards).

This would be directly beneficial for per-shard watermarks (FLINK-5697) to retain per-shard time characteristics after a reshard, and therefore can be seen as a prerequisite.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)