You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gyfora (via GitHub)" <gi...@apache.org> on 2023/06/08 10:53:21 UTC
[GitHub] [flink-kubernetes-operator] gyfora opened a new pull request, #615: [hotfix] Fix evaluation error during stabilization period
gyfora opened a new pull request, #615:
URL: https://github.com/apache/flink-kubernetes-operator/pull/615
## What is the purpose of the change
Fix a minor issue that happens during evaluating empty metrics during stabilization:
```
023-06-08 09:32:11,288 o.a.f.k.o.a.ScalingMetricCollector [INFO ][default/autoscaling-example] Job updated. Clearing metrics.
2023-06-08 09:32:11,288 o.a.f.k.o.a.ScalingMetricCollector [INFO ][default/autoscaling-example] Skipping metric collection during stabilization period until 2023-06-08T09:32:24.840Z
2023-06-08 09:32:11,288 o.a.f.k.o.a.JobAutoScalerImpl [ERROR][default/autoscaling-example] Error while scaling resource
java.util.NoSuchElementException
at java.base/java.util.TreeMap.key(TreeMap.java:1324)
at java.base/java.util.TreeMap.lastKey(TreeMap.java:296)
at java.base/java.util.Collections$UnmodifiableSortedMap.lastKey(Collections.java:1810)
at org.apache.flink.kubernetes.operator.autoscaler.ScalingMetricEvaluator.isProcessingBacklog(ScalingMetricEvaluator.java:90)
at org.apache.flink.kubernetes.operator.autoscaler.ScalingMetricEvaluator.evaluate(ScalingMetricEvaluator.java:68)
at org.apache.flink.kubernetes.operator.autoscaler.JobAutoScalerImpl.scale(JobAutoScalerImpl.java:117)
at org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:181)
at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:147)
at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)
at org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
## Brief change log
- Do not evaluate empty metrics
- Add test
## Verifying this change
Unit test added
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
- Core observer or reconciler logic that is regularly executed: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] pvary commented on a diff in pull request #615: [hotfix] Fix evaluation error during stabilization period
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #615:
URL: https://github.com/apache/flink-kubernetes-operator/pull/615#discussion_r1223968079
##########
flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/JobAutoScalerImpl.java:
##########
@@ -101,6 +101,11 @@ public boolean scale(FlinkResourceContext<? extends AbstractFlinkResource<?, ?>>
metricsCollector.updateMetrics(
resource, autoScalerInfo, ctx.getFlinkService(), conf);
+ if (collectedMetrics.getMetricHistory().isEmpty()) {
+ autoScalerInfo.replaceInKubernetes(kubernetesClient);
Review Comment:
nit: Does it worth to add a comment here as well, like:
```
// We have metrics yet, but we are not ready for scaling.
```
Do I understand correctly the intent of the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #615: [hotfix] Fix evaluation error during stabilization period
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora merged PR #615:
URL: https://github.com/apache/flink-kubernetes-operator/pull/615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #615: [hotfix] Fix evaluation error during stabilization period
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #615:
URL: https://github.com/apache/flink-kubernetes-operator/pull/615#discussion_r1224035631
##########
flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/JobAutoScalerImpl.java:
##########
@@ -101,6 +101,11 @@ public boolean scale(FlinkResourceContext<? extends AbstractFlinkResource<?, ?>>
metricsCollector.updateMetrics(
resource, autoScalerInfo, ctx.getFlinkService(), conf);
+ if (collectedMetrics.getMetricHistory().isEmpty()) {
+ autoScalerInfo.replaceInKubernetes(kubernetesClient);
Review Comment:
At this point we have no metrics, so there is nothing to do. What you commented actually happens a bit down in the code and has a similar comment :)
This piece of code was deleted in a previous commit I just restored it and added a test to make sure we dont accidentally remove/break it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org