You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "lynn1.zhang (Jira)" <ji...@apache.org> on 2021/03/22 05:42:00 UTC
[jira] [Updated] (FLINK-21833) TemporalRowTimeJoinOperator.java
will lead to the state expansion by short-life-cycle & huge RowData,
although config idle.state.retention.time
[ https://issues.apache.org/jira/browse/FLINK-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lynn1.zhang updated FLINK-21833:
--------------------------------
Description:
In our company cases, short-life-cycle & huge RowData will use this operator to join each other. Every key called session_id will be expired after 2min.
With idle.state.retention.time configuration, after 2min passed, the leftState and rightState will be cleaned up by the operator but nextLeftIndex & registeredTimer will be stored forever.
After running a day(about 20 million session_id in our cases ), the checkpoint operator will cause the job crash.
I have found the bug, and fixed it.
!image-2021-03-17-11-06-21-768.png!
was:
TemporalRowTimeJoinOperator feature will lead to unlimited data expansion, although configure idle.state.retention.time
I have found the bug, and fixed it.
!image-2021-03-17-11-06-21-768.png!
Summary: TemporalRowTimeJoinOperator.java will lead to the state expansion by short-life-cycle & huge RowData, although config idle.state.retention.time (was: TemporalRowTimeJoinOperator State Leak Although configure idle.state.retention.time)
> TemporalRowTimeJoinOperator.java will lead to the state expansion by short-life-cycle & huge RowData, although config idle.state.retention.time
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-21833
> URL: https://issues.apache.org/jira/browse/FLINK-21833
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Affects Versions: 1.12.2
> Reporter: lynn1.zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-03-17-11-06-21-768.png
>
>
> In our company cases, short-life-cycle & huge RowData will use this operator to join each other. Every key called session_id will be expired after 2min.
> With idle.state.retention.time configuration, after 2min passed, the leftState and rightState will be cleaned up by the operator but nextLeftIndex & registeredTimer will be stored forever.
> After running a day(about 20 million session_id in our cases ), the checkpoint operator will cause the job crash.
> I have found the bug, and fixed it.
> !image-2021-03-17-11-06-21-768.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)