You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/09/25 17:37:24 UTC

[GitHub] [flink-kubernetes-operator] tweise opened a new pull request, #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

tweise opened a new pull request, #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385

   ## What is the purpose of the change
   
   Automatically set jobId for Flink version < 1.16 to avoid conflicts in checkpoint path (and other jobId based effects).
   
   TODO:
   * Unit test
   * This breaks stateful upgrade if the CRD is not updated
   * We may want a better way to accommodate this in the status in the future rather than adding individual fields. Should we add a map that can be used for various purposes? 
   
   ## Brief change log
   
   * Set a jobId when no jobID is defined by user and Flink version is < 1.16.
   * Rotate jobId when upgrade mode is stateless
   
   ## Verifying this change
   
   TODO: This change added tests and can be verified as follows:
   
   Manual verification:
   * No jobId set in Flink config
     * Set upgrade mode to stateless. Trigger CR changes and check that every redeploy uses a new ID.
     * Set upgrade mode to last_state. Thrigger CR changes and check that 
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changes to the `CustomResourceDescriptors`: (**yes** / no)
     - Core observer or reconciler logic that is regularly executed: (yes / no)
   
   Field added to status and CRD needs to be updated for this change to work. If the CRD is not updated, the last state upgrade mode breaks.
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385#discussion_r979441596


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/crd/status/FlinkDeploymentStatus.java:
##########
@@ -53,4 +53,7 @@ public class FlinkDeploymentStatus extends CommonStatus<FlinkDeploymentSpec> {
 
     /** Information about the TaskManagers for the scale subresource. */
     private TaskManagerInfo taskManager;
+
+    /** The jobId, if it was set by the operator. */
+    private String overrideJobId;

Review Comment:
   All the current logic uses the jobStatus.jobId for savepointing, cancelling, observing etc, it would avoid a lot of complications :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385#discussion_r979441460


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/crd/status/FlinkDeploymentStatus.java:
##########
@@ -53,4 +53,7 @@ public class FlinkDeploymentStatus extends CommonStatus<FlinkDeploymentSpec> {
 
     /** Information about the TaskManagers for the scale subresource. */
     private TaskManagerInfo taskManager;
+
+    /** The jobId, if it was set by the operator. */
+    private String overrideJobId;

Review Comment:
   I think we should simply set it into the jobStatus.jobId and avoid introducing an extra field for it. 
   
   As long as the job is running we should assume that the jobStatus reflects the correct jobid
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385#discussion_r979763903


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java:
##########
@@ -195,6 +201,34 @@ protected void deploy(
                 relatedResource.getMetadata(), spec, deployConfig, kubernetesClient);
     }
 
+    private void setJobIdIfNecessary(
+            FlinkDeploymentSpec spec, FlinkDeploymentStatus status, Configuration deployConfig) {
+        // https://issues.apache.org/jira/browse/FLINK-19358
+        // https://issues.apache.org/jira/browse/FLINK-29109
+        if (spec.getFlinkVersion().isNewerVersionThan(FlinkVersion.v1_15)) {
+            return;
+        }
+
+        final JobID jobId;

Review Comment:
   We could use the String representation and set this directly from `status.getJobStatus().getJobId()` to get rid of the `else` branch later.



##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java:
##########
@@ -195,6 +201,34 @@ protected void deploy(
                 relatedResource.getMetadata(), spec, deployConfig, kubernetesClient);
     }
 
+    private void setJobIdIfNecessary(
+            FlinkDeploymentSpec spec, FlinkDeploymentStatus status, Configuration deployConfig) {
+        // https://issues.apache.org/jira/browse/FLINK-19358
+        // https://issues.apache.org/jira/browse/FLINK-29109
+        if (spec.getFlinkVersion().isNewerVersionThan(FlinkVersion.v1_15)) {
+            return;
+        }
+
+        final JobID jobId;
+        // generate jobId initially or rotate on every deployment when mode is stateless
+        if (status.getJobStatus().getJobId() == null
+                || spec.getJob().getUpgradeMode() == UpgradeMode.STATELESS) {
+            if (deployConfig.get(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID) != null) {
+                // user managed, don't touch
+                return;
+            }

Review Comment:
   We could move this before the `if` clause to simplify



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385#discussion_r979762326


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java:
##########
@@ -195,6 +201,34 @@ protected void deploy(
                 relatedResource.getMetadata(), spec, deployConfig, kubernetesClient);
     }
 
+    private void setJobIdIfNecessary(
+            FlinkDeploymentSpec spec, FlinkDeploymentStatus status, Configuration deployConfig) {
+        // https://issues.apache.org/jira/browse/FLINK-19358
+        // https://issues.apache.org/jira/browse/FLINK-29109
+        if (spec.getFlinkVersion().isNewerVersionThan(FlinkVersion.v1_15)) {
+            return;
+        }
+
+        final JobID jobId;
+        // generate jobId initially or rotate on every deployment when mode is stateless
+        if (status.getJobStatus().getJobId() == null
+                || spec.getJob().getUpgradeMode() == UpgradeMode.STATELESS) {
+            if (deployConfig.get(PipelineOptionsInternal.PIPELINE_FIXED_JOB_ID) != null) {
+                // user managed, don't touch
+                return;
+            }

Review Comment:
   We could move this check before the `if` clause to simplify



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] tweise commented on a diff in pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
tweise commented on code in PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385#discussion_r979497847


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/crd/status/FlinkDeploymentStatus.java:
##########
@@ -53,4 +53,7 @@ public class FlinkDeploymentStatus extends CommonStatus<FlinkDeploymentSpec> {
 
     /** Information about the TaskManagers for the scale subresource. */
     private TaskManagerInfo taskManager;
+
+    /** The jobId, if it was set by the operator. */
+    private String overrideJobId;

Review Comment:
   Reusing jobStatus.jobId is not perfect as we are using the same field for two different purposes. But it works pretty well and is better than adding another field to status for something that is going to be phased out when >=1.16 becomes the minimum version. PTAL



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink-kubernetes-operator] tweise merged pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)

Posted by GitBox <gi...@apache.org>.
tweise merged PR #385:
URL: https://github.com/apache/flink-kubernetes-operator/pull/385


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org