[jira] [Assigned] (FLINK-18235) Improve the checkpoint strategy for Python UDF execution

["Dian Fu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-18235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dian Fu reassigned FLINK-18235:
-------------------------------

    Assignee: Dian Fu

> Improve the checkpoint strategy for Python UDF execution
> --------------------------------------------------------
>
>                 Key: FLINK-18235
>                 URL: https://issues.apache.org/jira/browse/FLINK-18235
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / Python
>            Reporter: Dian Fu
>            Assignee: Dian Fu
>            Priority: Major
>              Labels: auto-deprioritized-major
>
> Currently, when a checkpoint is triggered for the Python operator, all the data buffered will be flushed to the Python worker to be processed. This will increase the overall checkpoint time in case there are a lot of elements buffered and Python UDF is slow. We should improve the checkpoint strategy to improve this. One way to implement this is to control the number of data buffered in the pipeline between Java/Python processes, similar to what [FLIP-183|https://cwiki.apache.org/confluence/display/FLINK/FLIP-183%3A+Dynamic+buffer+size+adjustment] does to control the number of data buffered in the network. We can also let users to config the checkpoint strategy if needed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)