You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gyfora (via GitHub)" <gi...@apache.org> on 2025/07/02 12:33:49 UTC

[PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

gyfora opened a new pull request, #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987

   ## What is the purpose of the change
   
   Fix a bug where if the snapshot where if periodic snapshot cleanup and snapshot CR are both enabled, upgrade save points can be accidentally deleted.
   
   ## Verifying this change
   
   new unit tests added
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
     - Core observer or reconciler logic that is regularly executed: yes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "salvalcantara (via GitHub)" <gi...@apache.org>.
salvalcantara commented on code in PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#discussion_r2181918327


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -409,25 +409,30 @@ protected void setUpgradeSavepointPath(
                         conf.get(KubernetesOperatorConfigOptions.OPERATOR_SAVEPOINT_FORMAT_TYPE)
                                 .name());
 
-        FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
-                conf,
-                ctx.getOperatorConfig(),
-                ctx.getKubernetesClient(),
-                ctx.getResource(),
-                savepointFormatType,
-                savepointLocation);
+        var snapshotCrOpt =
+                FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
+                        conf,
+                        ctx.getOperatorConfig(),
+                        ctx.getKubernetesClient(),
+                        ctx.getResource(),
+                        savepointFormatType,
+                        savepointLocation);
         var jobStatus = ctx.getResource().getStatus().getJobStatus();
         jobStatus.setUpgradeSavepointPath(savepointLocation);
 
-        // Register created savepoint in the now deprecated savepoint info and history
-        var savepoint =
-                new Savepoint(
-                        cancelTs.toEpochMilli(),
-                        savepointLocation,
-                        SnapshotTriggerType.UPGRADE,
-                        savepointFormatType,
-                        null);
-        jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        if (snapshotCrOpt.isEmpty()) {
+            // Register created savepoint in the now deprecated savepoint info and history
+            // only if snapshot CR was not created, otherwise it would be double recorded
+            // and disposed immediately
+            var savepoint =
+                    new Savepoint(
+                            cancelTs.toEpochMilli(),
+                            savepointLocation,
+                            SnapshotTriggerType.UPGRADE,
+                            savepointFormatType,
+                            null);
+            jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        }

Review Comment:
   @gyfora Maybe you could  use `isSnapshotResourceEnabled` here? Looks more explicit / less indirect to me. What if creation of the CR fails for example? Could that happen?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "salvalcantara (via GitHub)" <gi...@apache.org>.
salvalcantara commented on code in PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#discussion_r2181918327


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -409,25 +409,30 @@ protected void setUpgradeSavepointPath(
                         conf.get(KubernetesOperatorConfigOptions.OPERATOR_SAVEPOINT_FORMAT_TYPE)
                                 .name());
 
-        FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
-                conf,
-                ctx.getOperatorConfig(),
-                ctx.getKubernetesClient(),
-                ctx.getResource(),
-                savepointFormatType,
-                savepointLocation);
+        var snapshotCrOpt =
+                FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
+                        conf,
+                        ctx.getOperatorConfig(),
+                        ctx.getKubernetesClient(),
+                        ctx.getResource(),
+                        savepointFormatType,
+                        savepointLocation);
         var jobStatus = ctx.getResource().getStatus().getJobStatus();
         jobStatus.setUpgradeSavepointPath(savepointLocation);
 
-        // Register created savepoint in the now deprecated savepoint info and history
-        var savepoint =
-                new Savepoint(
-                        cancelTs.toEpochMilli(),
-                        savepointLocation,
-                        SnapshotTriggerType.UPGRADE,
-                        savepointFormatType,
-                        null);
-        jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        if (snapshotCrOpt.isEmpty()) {
+            // Register created savepoint in the now deprecated savepoint info and history
+            // only if snapshot CR was not created, otherwise it would be double recorded
+            // and disposed immediately
+            var savepoint =
+                    new Savepoint(
+                            cancelTs.toEpochMilli(),
+                            savepointLocation,
+                            SnapshotTriggerType.UPGRADE,
+                            savepointFormatType,
+                            null);
+            jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        }

Review Comment:
   :+1: Maybe you could  use `isSnapshotResourceEnabled` here? Looks more explicit / less indirect to me. What if creation of the CR fails for example? Could that happen?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "salvalcantara (via GitHub)" <gi...@apache.org>.
salvalcantara commented on code in PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#discussion_r2181918327


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -409,25 +409,30 @@ protected void setUpgradeSavepointPath(
                         conf.get(KubernetesOperatorConfigOptions.OPERATOR_SAVEPOINT_FORMAT_TYPE)
                                 .name());
 
-        FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
-                conf,
-                ctx.getOperatorConfig(),
-                ctx.getKubernetesClient(),
-                ctx.getResource(),
-                savepointFormatType,
-                savepointLocation);
+        var snapshotCrOpt =
+                FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
+                        conf,
+                        ctx.getOperatorConfig(),
+                        ctx.getKubernetesClient(),
+                        ctx.getResource(),
+                        savepointFormatType,
+                        savepointLocation);
         var jobStatus = ctx.getResource().getStatus().getJobStatus();
         jobStatus.setUpgradeSavepointPath(savepointLocation);
 
-        // Register created savepoint in the now deprecated savepoint info and history
-        var savepoint =
-                new Savepoint(
-                        cancelTs.toEpochMilli(),
-                        savepointLocation,
-                        SnapshotTriggerType.UPGRADE,
-                        savepointFormatType,
-                        null);
-        jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        if (snapshotCrOpt.isEmpty()) {
+            // Register created savepoint in the now deprecated savepoint info and history
+            // only if snapshot CR was not created, otherwise it would be double recorded
+            // and disposed immediately
+            var savepoint =
+                    new Savepoint(
+                            cancelTs.toEpochMilli(),
+                            savepointLocation,
+                            SnapshotTriggerType.UPGRADE,
+                            savepointFormatType,
+                            null);
+            jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        }

Review Comment:
   :+1: Maybe you could also use `isSnapshotResourceEnabled` instead? Looks more explicit / less indirect to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora merged PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#discussion_r2182038223


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -409,25 +409,30 @@ protected void setUpgradeSavepointPath(
                         conf.get(KubernetesOperatorConfigOptions.OPERATOR_SAVEPOINT_FORMAT_TYPE)
                                 .name());
 
-        FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
-                conf,
-                ctx.getOperatorConfig(),
-                ctx.getKubernetesClient(),
-                ctx.getResource(),
-                savepointFormatType,
-                savepointLocation);
+        var snapshotCrOpt =
+                FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
+                        conf,
+                        ctx.getOperatorConfig(),
+                        ctx.getKubernetesClient(),
+                        ctx.getResource(),
+                        savepointFormatType,
+                        savepointLocation);
         var jobStatus = ctx.getResource().getStatus().getJobStatus();
         jobStatus.setUpgradeSavepointPath(savepointLocation);
 
-        // Register created savepoint in the now deprecated savepoint info and history
-        var savepoint =
-                new Savepoint(
-                        cancelTs.toEpochMilli(),
-                        savepointLocation,
-                        SnapshotTriggerType.UPGRADE,
-                        savepointFormatType,
-                        null);
-        jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        if (snapshotCrOpt.isEmpty()) {
+            // Register created savepoint in the now deprecated savepoint info and history
+            // only if snapshot CR was not created, otherwise it would be double recorded
+            // and disposed immediately
+            var savepoint =
+                    new Savepoint(
+                            cancelTs.toEpochMilli(),
+                            savepointLocation,
+                            SnapshotTriggerType.UPGRADE,
+                            savepointFormatType,
+                            null);
+            jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        }

Review Comment:
   I don't think that it could fail without an error and even if that were the case it would probably be better then to record this info still. The bug was caused by the information recorded in 2 places and this interfering with the savepoint cleanup logic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "salvalcantara (via GitHub)" <gi...@apache.org>.
salvalcantara commented on PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#issuecomment-3030948470

   Thanks @gyfora @gaborgsomogyi!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38033] Fix accidental upgrade snapshot dispose bug [flink-kubernetes-operator]

Posted by "salvalcantara (via GitHub)" <gi...@apache.org>.
salvalcantara commented on code in PR #987:
URL: https://github.com/apache/flink-kubernetes-operator/pull/987#discussion_r2181918327


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java:
##########
@@ -409,25 +409,30 @@ protected void setUpgradeSavepointPath(
                         conf.get(KubernetesOperatorConfigOptions.OPERATOR_SAVEPOINT_FORMAT_TYPE)
                                 .name());
 
-        FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
-                conf,
-                ctx.getOperatorConfig(),
-                ctx.getKubernetesClient(),
-                ctx.getResource(),
-                savepointFormatType,
-                savepointLocation);
+        var snapshotCrOpt =
+                FlinkStateSnapshotUtils.createUpgradeSnapshotResource(
+                        conf,
+                        ctx.getOperatorConfig(),
+                        ctx.getKubernetesClient(),
+                        ctx.getResource(),
+                        savepointFormatType,
+                        savepointLocation);
         var jobStatus = ctx.getResource().getStatus().getJobStatus();
         jobStatus.setUpgradeSavepointPath(savepointLocation);
 
-        // Register created savepoint in the now deprecated savepoint info and history
-        var savepoint =
-                new Savepoint(
-                        cancelTs.toEpochMilli(),
-                        savepointLocation,
-                        SnapshotTriggerType.UPGRADE,
-                        savepointFormatType,
-                        null);
-        jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        if (snapshotCrOpt.isEmpty()) {
+            // Register created savepoint in the now deprecated savepoint info and history
+            // only if snapshot CR was not created, otherwise it would be double recorded
+            // and disposed immediately
+            var savepoint =
+                    new Savepoint(
+                            cancelTs.toEpochMilli(),
+                            savepointLocation,
+                            SnapshotTriggerType.UPGRADE,
+                            savepointFormatType,
+                            null);
+            jobStatus.getSavepointInfo().updateLastSavepoint(savepoint);
+        }

Review Comment:
   :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org