You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dennis-Mircea (via GitHub)" <gi...@apache.org> on 2026/10/20 15:29:05 UTC

[PR] [FLINK-41050] Canary health checks revert canary metadata changes and restart a healthy operator [flink-kubernetes-operator]

Dennis-Mircea opened a new pull request, #1227:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1227

   ## What is the purpose of the change
   
   This PR makes the canary update safe against concurrent changes, retries failed updates instead of failing the health check, and stops tracking canaries the operator no longer owns.
   
   ## Brief change log
   
   - The canary spec update reads the latest canary from the API server and updates it with `update()`, instead of calling `replace()` on the cached copy.
   - The first update runs after `ConcurrentHashMap.compute`, not inside it, avoiding failing the main controller reconcile cycle (mainly because if the bump gets a 404 or finds the label gone, it calls `canaryResources.remove(resourceID, crs)` on that same key, which will cause an exception that will be propagated upwards) or holding a lock that may be produced by network I/O.
   - Each check verifies the last successful update. When the update fails, the check stays healthy if that update was reconciled, and logs a warning. After `kubernetes.operator.health.canary.resource.retry.max.attempts` (new, default 3) consecutive failed updates, the canary is reported unhealthy. A value `<= 0` disables this.
   - The canary state is dropped when the canary is not found (404), when the read canary no longer has the canary label, when a resource without the label is reconciled, and when its namespace is removed from the watched namespaces (`FlinkOperator.handleNamespaceChanges` calls the new `CanaryResourceManager.retainNamespaces`).
   - The health check accepts a reconciled generation greater than or equal to the updated one, and the logs name the canary and the reason.
   - The class JavaDoc was updated as well, because previously it was wrongly displaying this: `/** Logic encapsulating canary tests. */`.
   
   ## Verifying this change
   The change can be verified by the new tests added.
   
   The change was also verified manually on minikube and the stacktraces and scenarios presented in the JIRA are no longer reproducible.
   
   NOTE: Continuous concurrent metadata writes cause an occasional 409 on the update. It is retried on the next check, and the canary stays healthy.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
     - Core observer or reconciler logic that is regularly executed: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
   "Generated-by" line. See the ASF Generative Tooling Guidance for details:
   https://www.apache.org/legal/generative-tooling.html
   
   You are responsible for the quality and correctness of every change in this PR
   regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
   AGENTS.md for the full guidance.
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name and Version]
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41050] Canary health checks revert canary metadata changes and restart a healthy operator [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora merged PR #1227:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1227


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41050] Canary health checks revert canary metadata changes and restart a healthy operator [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #1227:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1227#discussion_r4401743432


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/KubernetesOperatorConfigOptions.java:
##########
@@ -715,6 +715,14 @@ public static String operatorConfigKey(String key) {
                     .withDescription(
                             "Allowed max time between spec update and reconciliation for canary resources.");
 
+    @Documentation.Section(SECTION_ADVANCED_STARTUP)
+    public static final ConfigOption<Integer> CANARY_RESOURCE_RETRY_MAX_ATTEMPTS =
+            operatorConfig("health.canary.resource.retry.max.attempts")
+                    .intType()
+                    .defaultValue(3)

Review Comment:
   I would really prefer to hardcode this to some reasonable number (3 sounds good ) and not make it configurable. feels kind of useless config, lets save the space :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-41050] Canary health checks revert canary metadata changes and restart a healthy operator [flink-kubernetes-operator]

Posted by "Dennis-Mircea (via GitHub)" <gi...@apache.org>.
Dennis-Mircea commented on code in PR #1227:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1227#discussion_r4402505003


##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/KubernetesOperatorConfigOptions.java:
##########
@@ -715,6 +715,14 @@ public static String operatorConfigKey(String key) {
                     .withDescription(
                             "Allowed max time between spec update and reconciliation for canary resources.");
 
+    @Documentation.Section(SECTION_ADVANCED_STARTUP)
+    public static final ConfigOption<Integer> CANARY_RESOURCE_RETRY_MAX_ATTEMPTS =
+            operatorConfig("health.canary.resource.retry.max.attempts")
+                    .intType()
+                    .defaultValue(3)

Review Comment:
   Ok, makes sense. I've updated the PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org