[jira] [Created] (FLINK-40467) Flink 2.3 Application Mode managed by the kubernetes operator can start without submitting the new job after savepoint redeploy

["Josh Keegan (Jira)" <ji...@apache.org>]

Josh Keegan created FLINK-40467:
-----------------------------------

             Summary: Flink 2.3 Application Mode managed by the kubernetes operator can start without submitting the new job after savepoint redeploy
                 Key: FLINK-40467
                 URL: https://issues.apache.org/jira/browse/FLINK-40467
             Project: Flink
          Issue Type: Bug
          Components: Kubernetes Operator
    Affects Versions: 2.3.0
            Reporter: Josh Keegan


With Flink 2.3 Application Mode, we have seen savepoint upgrades complete successfully but the replacement application cluster sometimes starts without submitting the new job.

The operator assigns a new Job ID, while the Application ID remains the same as the previous deployment. The JobManagers are healthy and REST is available, but the operator reports {{{}MISSING / Job Not Found{}}}. Looking in the UI, the previous application shows as {{FINISHED}} with no jobs.

Looking into this, it seems that the operator replaces the Job ID during a savepoint redeployment since the old Job has been stopped. The same logic doesn't exist for the new Application ID.

We worked around this by setting the Application ID to the Job ID assigned by the operator in a startup script for our docker image. That way, whenever the operator rotates the Job ID, our Application ID is also rotated. Since doing this approx 1 week ago, we've had no issues.

It looks like the operator should manage the Application ID alongside the Job ID for new Application Mode executions. I think our fix of setting it to be the same as the Job ID is fine for single Job deployments, but I don't know if more complex logic is required for multiple Jobs.


Using Flink 2.3 and flink-kubernetes-operator 1.15.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)