You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "fqshopify (via GitHub)" <gi...@apache.org> on 2026/07/10 16:27:47 UTC
[PR] [FLINK-XXX] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
fqshopify opened a new pull request, #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159
## What is the purpose of the change
When a Blue/Green transition is aborted, `abortDeployment` rolled the state back to `ACTIVE_BLUE` but left `deploymentReadyTimestamp` set to the value from the aborted transition (only the finalize paths reset it). On the **next** deployment, the delete-Blue gate in `shouldWeDelete` computes `deletionTimestamp = staleReadyTimestamp + deletionDelay`, which is already in the past, so Blue is deleted the instant Green first reports `RUNNING` - skipping the deletion delay entirely and causing downtime.
This pull request makes `abortDeployment` reset the transition markers the same way the finalize paths do, so a redeploy after an abort re-arms the delete-Blue deletion delay from scratch.
## Brief change log
- `abortDeployment` now resets `deploymentReadyTimestamp` to 0 on abort, so the next transition re-arms the delete-Blue delay instead of treating it as long expired.
- `abortDeployment` now also resets `abortTimestamp` to 0, making the abort teardown symmetric with `finalizeBlueGreenDeployment` / `finalizeSuspendedDeployment` (which already reset `deploymentReadyTimestamp`, `abortTimestamp`, and `savepointTriggerId`).
- Added a regression test exercising the full abort => redeploy lifecycle in `SAVEPOINT` upgrade mode.
## Verifying this change
This change added tests and can be verified using `mvn clean verify`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: **no**
- Core observer or reconciler logic that is regularly executed: **yes** (the Blue/Green abort path in the reconciler)
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable** (bug fix)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "Dennis-Mircea (via GitHub)" <gi...@apache.org>.
Dennis-Mircea commented on code in PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159#discussion_r3561417831
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/bluegreen/BlueGreenDeploymentService.java:
##########
@@ -686,6 +686,8 @@ private UpdateControl<FlinkBlueGreenDeployment> abortDeployment(
getPreviousState(nextState, context.getDeployments());
context.getDeploymentStatus().setBlueGreenState(previousState);
context.getDeploymentStatus().setSavepointTriggerId(null);
+ context.getDeploymentStatus().setDeploymentReadyTimestamp(millisToInstantStr(0));
+ context.getDeploymentStatus().setAbortTimestamp(millisToInstantStr(0));
Review Comment:
These three resets (+ `savepointTriggerId` above) are now duplicated in `finalizeBlueGreenDeployment`, `finalizeSuspendedDeployment`, and here. Could you extract a `resetTransitionMarkers(status)` helper and call it from all three, so they can't diverge again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "drossos (via GitHub)" <gi...@apache.org>.
drossos commented on code in PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159#discussion_r3561412201
##########
flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentControllerTest.java:
##########
Review Comment:
I believe this breaks now because it is still assuming the previous behaviour where after abort timestamp won't be zero. Should probably do:
```suggestion
assertEquals(0, instantStrToMillis(rs.reconciledStatus.getAbortTimestamp()));
```
That allows the test to pass with this new timer reset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "fqshopify (via GitHub)" <gi...@apache.org>.
fqshopify commented on code in PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159#discussion_r3580520650
##########
flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentControllerTest.java:
##########
Review Comment:
Fixed
##########
flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentControllerTest.java:
##########
@@ -558,6 +558,198 @@ public void verifyFailureDuringTransition(FlinkVersion flinkVersion) throws Exce
testTransitionToGreen(rs, customValue, null);
}
+ /** Regression test: an aborted transition must reset the delete-Blue deletion delay. */
+ @ParameterizedTest
+ @MethodSource("org.apache.flink.kubernetes.operator.TestUtils#flinkVersions")
+ public void verifyReadyTimestampResetOnAbortPreventsPrematureBlueDelete(
+ FlinkVersion flinkVersion) throws Exception {
+ var blueGreenDeployment =
+ buildSessionCluster(
+ TEST_DEPLOYMENT_NAME,
+ TEST_NAMESPACE,
+ flinkVersion,
+ null,
+ UpgradeMode.SAVEPOINT);
+
+ // Abort grace long enough to let Green be marked ready before any abort fires;
+ // a non-trivial deletion delay so the delete-Blue wait is observable.
+ var abortGracePeriodMs = 1500;
+ var deletionDelayMs = 1000;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "fqshopify (via GitHub)" <gi...@apache.org>.
fqshopify commented on code in PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159#discussion_r3580522814
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/bluegreen/BlueGreenDeploymentService.java:
##########
@@ -686,6 +686,8 @@ private UpdateControl<FlinkBlueGreenDeployment> abortDeployment(
getPreviousState(nextState, context.getDeployments());
context.getDeploymentStatus().setBlueGreenState(previousState);
context.getDeploymentStatus().setSavepointTriggerId(null);
+ context.getDeploymentStatus().setDeploymentReadyTimestamp(millisToInstantStr(0));
+ context.getDeploymentStatus().setAbortTimestamp(millisToInstantStr(0));
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora merged PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40119] Reset deploymentReadyTimestamp on abort to prevent premature blue delete on next deploy [flink-kubernetes-operator]
Posted by "Dennis-Mircea (via GitHub)" <gi...@apache.org>.
Dennis-Mircea commented on code in PR #1159:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1159#discussion_r3561402528
##########
flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentControllerTest.java:
##########
@@ -558,6 +558,198 @@ public void verifyFailureDuringTransition(FlinkVersion flinkVersion) throws Exce
testTransitionToGreen(rs, customValue, null);
}
+ /** Regression test: an aborted transition must reset the delete-Blue deletion delay. */
+ @ParameterizedTest
+ @MethodSource("org.apache.flink.kubernetes.operator.TestUtils#flinkVersions")
+ public void verifyReadyTimestampResetOnAbortPreventsPrematureBlueDelete(
+ FlinkVersion flinkVersion) throws Exception {
+ var blueGreenDeployment =
+ buildSessionCluster(
+ TEST_DEPLOYMENT_NAME,
+ TEST_NAMESPACE,
+ flinkVersion,
+ null,
+ UpgradeMode.SAVEPOINT);
+
+ // Abort grace long enough to let Green be marked ready before any abort fires;
+ // a non-trivial deletion delay so the delete-Blue wait is observable.
+ var abortGracePeriodMs = 1500;
+ var deletionDelayMs = 1000;
Review Comment:
Consider using `await()` instead of these long sleep intervals.
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/bluegreen/BlueGreenDeploymentService.java:
##########
@@ -686,6 +686,8 @@ private UpdateControl<FlinkBlueGreenDeployment> abortDeployment(
getPreviousState(nextState, context.getDeployments());
context.getDeploymentStatus().setBlueGreenState(previousState);
context.getDeploymentStatus().setSavepointTriggerId(null);
+ context.getDeploymentStatus().setDeploymentReadyTimestamp(millisToInstantStr(0));
+ context.getDeploymentStatus().setAbortTimestamp(millisToInstantStr(0));
Review Comment:
These three resets (+ `savepointTriggerId` above) are now duplicated in `finalizeBlueGreenDeployment`, `finalizeSuspendedDeployment`, and here. Could we extract a `resetTransitionMarkers(status)` helper and call it from all three, so they can't diverge again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org