You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Francisco Morillo <fm...@gmail.com> on 2026/06/01 14:47:25 UTC

[DISCUSS] FLIP-586: Introduce the upsert-kinesis Connector

 I'd like to propose introducing an upsert-kinesis connector for Flink SQL,
  following the same pattern as upsert-kafka (FLIP-149).

  Currently, the kinesis connector only supports append (INSERT-only)
streams.
  This means GROUP BY, deduplication, and streaming joins cannot write
results
  to Kinesis Data Streams — the job fails with "Table sink doesn't support
  consuming update and delete changes."

  The proposal adds a separate 'upsert-kinesis' connector that:
  - Requires a PRIMARY KEY definition
  - Accepts INSERT, UPDATE_AFTER, and DELETE changelog events
  - Uses primary key fields as the Kinesis partition key
  - Writes empty-payload tombstones for deletes
  - Reuses the existing KinesisStreamsSink/Source (no code duplication)

  Full proposal:
https://docs.google.com/document/d/1x5BmeUKC_Xi-2opdYUJmBZCtB9sj4DxPidvhy5PW4hE/edit?usp=sharing
  Draft implementation:
https://github.com/apache/flink-connector-aws/pull/249

  Looking forward to your feedback.

  Best regards,
Francisco Morillo