[jira] [Comment Edited] (FLINK-29959) Use optimistic locking when patching resource status

["Gyula Fora (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-29959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633948#comment-17633948 ] 

Gyula Fora edited comment on FLINK-29959 at 11/14/22 5:10 PM:
--------------------------------------------------------------

There are 2 main problems that we target here:
 # Status updates by zombie operators who has last lost leadership but not realized/dead yet.
 # Stale status received when a new leader starts

*Why would these happen?*

Zombie operator:
It could in theory happen that an operator loses leadership in a middle of reconciliation due to a very long GC pause (or some network issue or whatever) and the current CR reconcile loop continues while the new leader already started to reconcile this resource. This is very unlikely but can happen with leader election and a standby operator. In these cases we don't want to allow the old operator who lost leadership to be able to make any status updates. The new logic guarantees that if the new leader made any status update the old would never be able to do so again.

Stale status:
When the new leader starts processing (if it was on standby) there is no guarantee that the status/spec reconciled at the first time is up to date. This can happen because due to some unlucky cache update timing or even a zombie operator submitting late status updates. The current operator logic very much relies on seeing the last status otherwise we can have some very weird cornercases that would definitely cause problems for the resources.

*How the new logic tackles this in a safe way*

What the new logic does is that it basically only allows status updates to go through when the operator has the latest status information. So it's sort of a locking on the current status. If anyone else changed the status in the meantime, we simply throw an error and retrigger the reconciliation. This is actually safe to do as the operator reconcile logic already runs with the assumption that the operator can fail at any time before status update, and we always use the status as a "write-ahead-log" of the actions we are taking. In these cases zombie operators who have already lost leadership would never reconcile again (the leader election guarantees that), and in other cases this would give us the latest version of the resource.


was (Author: gyfora):
There are 2 main problems that we target here:
 # Status updates by zombie operators who has last lost leadership but not realized/dead yet.
 # Stale status received when a new leader starts

*Why would these happen?*

Zombie operator:
It could in theory happen that an operator loses leadership in a middle of reconciliation due to a very long GC pause (or some network issue or whatever) and the current CR reconcile loop continues while the new leader already started to reconcile this resource. This is very unlikely but can happen with leader election and a standby operator. In these cases we don't want to allow the old operator who lost leadership to be able to make any status updates. The new logic guarantees that if the new leader made any status update the old would never be able to do so again.

Stale status:
When the new leader starts processing (if it was on standby) there is no guarantee that the status/spec reconciled at the first time is up to date. This can happen because due to some unlucky cache update timing or even a zombie operator submitting late status updates. The current operator logic very much relies on seeing the last status otherwise we can have some very weird cornercases that would definitely cause problems for the resources.

*How the new logic tackles this in a safe way*

What the new logic does is that it basically only allows status updates to go through when the operator has the latest status information. So it's sort of a locking on the current status. If anyone else changed the status in the meantime, we simply throw an error and retrigger the reconciliation. This is actually safe to do as the operator reconcile logic already runs with the assumption that the operator can fail at any time before status update, and we always use the status as a "write-ahead-log" of the actions we are taking. In these cases zombie operators who have already lost leadership would never reconcile again (the leader election guarantees that), and in other cases this would give us the latest version of the resource.

Additional notes:
When the upstream PR is open, I will loop in Attila from the JOSDK team to help review it. I was discussing these cases with him previously and we came to this solution.

> Use optimistic locking when patching resource status
> ----------------------------------------------------
>
>                 Key: FLINK-29959
>                 URL: https://issues.apache.org/jira/browse/FLINK-29959
>             Project: Flink
>          Issue Type: Bug
>          Components: Kubernetes Operator
>            Reporter: Gyula Fora
>            Assignee: Gyula Fora
>            Priority: Critical
>
> The operator currently does not use optimistic locking on the CR when patching status. This worked because we always wanted to overwrite the status.
> With leader election and potentially two operators running at the same time, we are now exposed to some race conditions that were not previously present with the status update logic.
> To ensure that the operator always sees the latest status we should change our logic to optimistic locking with retries. If we get a lock error (resource updated) we check if only the spec changed and then retry locking on the new version.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)