[jira] [Updated] (FLINK-11618) [state] Refactor operator state repartition mechanism

["Yun Tang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-11618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yun Tang updated FLINK-11618:
-----------------------------
    Description: 
Currently we have state assignment strategy of operator state below:
 * When parallelism not changed:
 ** If we only have even-split redistributed state, state assignment would try to keep as the same as previously (actually not always the same).
 ** If we have union redistributed state, all the operator state would be redistributed as the new state assignment.
 * When parallelism changed:
 ** all the operator state would be redistributed as the new state assignment.

There existed two problems *when parallelism not changed*:
 # If we only have even-split redistributed state, current implementation actually cannot ensure state assignment to keep as the same as previously. This is because current {{StateAssignmentOperation#collectPartitionableStates}} would repartition {{managedOperatorStates}} without subtask-index information. Take an example, if we have a operator-state with parallelism as 2, and subtask-0's managed-operatorstate is empty while subtask-1 not. Although new parallelism still keeps as 2, after {{StateAssignmentOperation#collectPartitionableStates}} and state assigned, subtask-0 would be assigned the managed-operatorstate while subtask-1 got none.
 # We should only redistribute union state and not touch the even-split state. Redistribute even-split state would cause unexpected behavior after {{RestartPipelinedRegionStrategy}} supported to restore state.

We should fix the above two problems and this issue is a prerequisite of FLINK-10712 and FLINK-10713 .

  was:
Currently we have state assignment strategy of operator state below:
 * When parallelism not changed:
 ** If we only have even-split redistributed state, state assignment would try to keep as the same as previously (actually not always the same).
 ** If we have union redistributed state, all the operator state would be redistributed as the new state assignment.
 * When parallelism changed:
 ** all the operator state would be redistributed as the new state assignment.

There existed two problems *when parallelism not changed*:
 # If we only have even-split redistributed state, current implementation actually cannot ensure state assignment to keep as the same as previously. This is because current {{StateAssignmentOperation#collectPartitionableStates}} would repartition {{managedOperatorStates}} without subtask-index information. Take and example, if we have a operator-state with parallelism as 2, and subtask-0's managed-operatorstate is empty while subtask-1 not. Although new parallelism still keeps as 2, after {{StateAssignmentOperation#collectPartitionableStates}}, subtask-0 would be assigned the managed-operatorstate but subtask-1 get none.
 # We should only redistribute union state and not touch the even-split state. Redistribute even-split state would cause unexpected behavior after {{RestartPipelinedRegionStrategy}} supported to restore state.

We should fix the above two problems and this issue is a prerequisite of FLINK-10712 and FLINK-10713 .


> [state] Refactor operator state repartition mechanism
> -----------------------------------------------------
>
>                 Key: FLINK-11618
>                 URL: https://issues.apache.org/jira/browse/FLINK-11618
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.7.0
>            Reporter: Yun Tang
>            Assignee: Yun Tang
>            Priority: Major
>             Fix For: 1.8.0
>
>
> Currently we have state assignment strategy of operator state below:
>  * When parallelism not changed:
>  ** If we only have even-split redistributed state, state assignment would try to keep as the same as previously (actually not always the same).
>  ** If we have union redistributed state, all the operator state would be redistributed as the new state assignment.
>  * When parallelism changed:
>  ** all the operator state would be redistributed as the new state assignment.
> There existed two problems *when parallelism not changed*:
>  # If we only have even-split redistributed state, current implementation actually cannot ensure state assignment to keep as the same as previously. This is because current {{StateAssignmentOperation#collectPartitionableStates}} would repartition {{managedOperatorStates}} without subtask-index information. Take an example, if we have a operator-state with parallelism as 2, and subtask-0's managed-operatorstate is empty while subtask-1 not. Although new parallelism still keeps as 2, after {{StateAssignmentOperation#collectPartitionableStates}} and state assigned, subtask-0 would be assigned the managed-operatorstate while subtask-1 got none.
>  # We should only redistribute union state and not touch the even-split state. Redistribute even-split state would cause unexpected behavior after {{RestartPipelinedRegionStrategy}} supported to restore state.
> We should fix the above two problems and this issue is a prerequisite of FLINK-10712 and FLINK-10713 .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)