You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Gustavo de Morais (Jira)" <ji...@apache.org> on 2025/09/22 07:18:00 UTC
[jira] [Updated] (FLINK-37859) FLIP-516: Streaming Multi-Way Join Optimization
[ https://issues.apache.org/jira/browse/FLINK-37859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gustavo de Morais updated FLINK-37859:
--------------------------------------
Description:
We currently always rely on a chain of binary joins operators for multiple non-temporal regular joins in a flink streaming job. This often generates a lot of intermediate state which considerably increases the state size and leads to decrease performance and performance degradation.
[FLIP-516|https://cwiki.apache.org/confluence/display/FLINK/FLIP-516%3A+Multi-Way+Join+Operator] proposes and implements an operator that performs a join accross N inputs with zero intermediate state and better performance for pipelines with records amplification. This is a challenging task since real-world implementations of a MultiJoin operator for a changelog stream processor [are not known and the algorithm itself is complicated.|https://issues.apache.org/jira/browse/SPARK-2215?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=14038513]
was:
We currently always rely on a chain of binary joins operators for multiple non-temporal regular joins in a flink streaming job. This often generates a lot of intermediate state which considerably increases the state size and leads to decrease performance and performance degradation.
[FLIP-516|https://cwiki.apache.org/confluence/display/FLINK/FLIP-516%3A+Multi-Way+Join+Operator] proposes and implements an operator that performs a join accross N inputs with zero intermediate state and better performance for pipelines with records amplification.
> FLIP-516: Streaming Multi-Way Join Optimization
> -----------------------------------------------
>
> Key: FLINK-37859
> URL: https://issues.apache.org/jira/browse/FLINK-37859
> Project: Flink
> Issue Type: New Feature
> Reporter: dalongliu
> Assignee: Gustavo de Morais
> Priority: Major
>
> We currently always rely on a chain of binary joins operators for multiple non-temporal regular joins in a flink streaming job. This often generates a lot of intermediate state which considerably increases the state size and leads to decrease performance and performance degradation.
>
> [FLIP-516|https://cwiki.apache.org/confluence/display/FLINK/FLIP-516%3A+Multi-Way+Join+Operator] proposes and implements an operator that performs a join accross N inputs with zero intermediate state and better performance for pipelines with records amplification. This is a challenging task since real-world implementations of a MultiJoin operator for a changelog stream processor [are not known and the algorithm itself is complicated.|https://issues.apache.org/jira/browse/SPARK-2215?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=14038513]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)