[jira] [Closed] (FLINK-27559) Some question about flink operator state

["Yun Tang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-27559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yun Tang closed FLINK-27559.
----------------------------
    Resolution: Information Provided

> Some question about flink operator state
> ----------------------------------------
>
>                 Key: FLINK-27559
>                 URL: https://issues.apache.org/jira/browse/FLINK-27559
>             Project: Flink
>          Issue Type: New Feature
>         Environment: Flink 1.14.4
>            Reporter: Underwood
>            Priority: Major
>
> I hope to get two answers to Flink's maintenance status:
>  
> 1. Does custompartition support saving status? In my usage scenario, the partition strategy is dynamically adjusted, which depends on the data in datastream. I hope to make different partition strategies according to different data conditions.
>  
> For a simple example, I want the first 100 pieces of data in datastream to be range partitioned and the rest of the data to be hash partitioned. At this time, I may need a count to identify the number of pieces of data that have been processed. However, in custompartition, this is only a local variable, so there seem to be two problems: declaring variables in this way can only be used in single concurrency, and it seems that they cannot be counted across slots; In this way, the count data will be lost during fault recovery.
>  
> Although Flink already has operator state and key value state, custompartition is not an operator, so I don't think it can solve this problem through state. I've considered introducing a zookeeper to save the state, but the introduction of new components will make the system bloated. I don't know whether there is a better way to solve this problem.
>  
> 2. How to make multiple operators share the same state, and even all parallel subtasks of different operators share the same state?
>  
> For a simple example, my stream processing is divided into four stages: source - > mapa - > mapb - > sink. I hope to have a status count to count the total amount of data processed by all operators. For example, if the source receives one piece of data, then count + 1 when mapa is processed and count + 1 when mapb is processed. Finally, after this piece of data is processed, the value of count is 2.
>  
> I don't know if there is such a state saving mechanism in Flink, which can meet my scenario and recover from failure at the same time. At present, we can still think of using zookeeper. I don't know if there is a better way.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)