[jira] [Updated] (FLINK-12294) Kafka connector, work with grouping partitions

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-12294:
-----------------------------------
    Labels: auto-deprioritized-major performance stale-minor  (was: auto-deprioritized-major performance)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.


> Kafka connector, work with grouping partitions
> ----------------------------------------------
>
>                 Key: FLINK-12294
>                 URL: https://issues.apache.org/jira/browse/FLINK-12294
>             Project: Flink
>          Issue Type: New Feature
>          Components: API / DataStream, Connectors / Kafka, Runtime / Task
>            Reporter: Sergey
>            Priority: Minor
>              Labels: auto-deprioritized-major, performance, stale-minor
>         Attachments: KeyGroupAssigner.java, KeyGroupRangeAssignment.java
>
>
> Additional flag (with default false value) controlling whether topic partitions already grouped by the key. Exclude unnecessary shuffle/resorting operation when this parameter set to true. As an example, say we have client's payment transaction in a kafka topic. We grouping by clientId (transaction with the same clientId goes to one kafka topic partition) and the task is to find max transaction per client in sliding windows. In terms of map\reduce there is no needs to shuffle data between all topic consumers, may be it`s worth to do within each consumer to gain some speedup due to increasing number of executors within each partition data. With N messages (in partition) instead of N*ln(N) (current realization with shuffle/resorting) it will be just N operations. For windows with thousands events - the tenfold gain of execution speed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)