You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Stephan Ewen (Jira)" <ji...@apache.org> on 2020/01/19 22:45:00 UTC

[jira] [Created] (FLINK-15670) Provide a Kafka Source/Sink pair that aligns Kafka's Partitions and Flink's KeyGroups

Stephan Ewen created FLINK-15670:
------------------------------------

             Summary: Provide a Kafka Source/Sink pair that aligns Kafka's Partitions and Flink's KeyGroups
                 Key: FLINK-15670
                 URL: https://issues.apache.org/jira/browse/FLINK-15670
             Project: Flink
          Issue Type: Improvement
          Components: API / DataStream, Connectors / Kafka
            Reporter: Stephan Ewen
             Fix For: 1.11.0


This Source/Sink pair would serve two purposes:

1. You can read topics that are already partitioned by key and process them without partitioning them again (avoid shuffles)

2. You can use this to shuffle through Kafka, thereby decomposing the job into smaller jobs and independent pipelined regions that fail over independently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)