You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2023/03/17 18:16:47 UTC
[GitHub] [flink-kubernetes-operator] mxm opened a new pull request, #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
mxm opened a new pull request, #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549
Until we move to using the upcoming Rescale API which recycles pods, we need to be mindful with how many deployments we scale at the same time because each of them is going to give up all its pods and require the new number of required pods.
This can cause churn in the cluster and temporary lead to "unallocatable" pods which triggers the k8s cluster autoscaler to add more cluster nodes. That is often not desirable because the actual required resources after the scalings have been settled, are lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#issuecomment-1477998088
The logic was implemented in the global context because it handles logic across pipelines, something which the pipeline-specific ScalingExecutor cannot do. The logic can be used independently of the autoscaler implementation. I tried to implement the logic in the autoscaler directly but the the current design of operator / autoscaler doesn't easily allow this because the autoscaler logic is pipeline-specific without state across pipelines. If you have any ideas, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#discussion_r1142244157
##########
docs/layouts/shortcodes/generated/dynamic_section.html:
##########
@@ -86,6 +86,12 @@
<td>Duration</td>
<td>Interval at which periodic savepoints will be triggered. The triggering schedule is not guaranteed, savepoints will be triggered as part of the regular reconcile loop.</td>
</tr>
+ <tr>
+ <td><h5>kubernetes.operator.rescaling.cluster-cooldown</h5></td>
+ <td style="word-wrap: break-word;">1 min</td>
+ <td>Duration</td>
Review Comment:
I personally would not enforce any cross deployment timeout / cooldown by default. I don't think it's intuitive why something on one deployment should have an effect on others. Also this is specific to EKS not a general problem.
While the lack of this config may cause problems on EKS over time, it causes immediate weird behaviour on every operator that is using the autoscaler for multiple jobs. Imagine having 60 jobs, it will take an hour to scale once. Also some jobs may never be scaled because there is no faire queing etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm closed pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm closed pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
URL: https://github.com/apache/flink-kubernetes-operator/pull/549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#issuecomment-1488473282
Based on our offline discussions, this feature may not be necessary anymore. Would you like to keep this open or should we revisit at a future time ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on a diff in pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#discussion_r1141846057
##########
docs/layouts/shortcodes/generated/dynamic_section.html:
##########
@@ -86,6 +86,12 @@
<td>Duration</td>
<td>Interval at which periodic savepoints will be triggered. The triggering schedule is not guaranteed, savepoints will be triggered as part of the regular reconcile loop.</td>
</tr>
+ <tr>
+ <td><h5>kubernetes.operator.rescaling.cluster-cooldown</h5></td>
+ <td style="word-wrap: break-word;">1 min</td>
+ <td>Duration</td>
Review Comment:
This one is tricky. I would rather like to be conservative here because concurrent scaling operations can create a lot of unexpected churn in the cluster. Users have seen cost increase from resource spikes which can paralyze the entire cluster and lead to adding more k8s nodes (if the cluster autoscaler is active). Users may always change this default.
Beyond this, we may add something like a concurrency parameter which allows to define how many pipelines are allowed to scale before the cool down. Additionally, checking the actual allocatable resources before scaling is something we need to look into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#discussion_r1140649362
##########
docs/layouts/shortcodes/generated/dynamic_section.html:
##########
@@ -86,6 +86,12 @@
<td>Duration</td>
<td>Interval at which periodic savepoints will be triggered. The triggering schedule is not guaranteed, savepoints will be triggered as part of the regular reconcile loop.</td>
</tr>
+ <tr>
+ <td><h5>kubernetes.operator.rescaling.cluster-cooldown</h5></td>
+ <td style="word-wrap: break-word;">1 min</td>
+ <td>Duration</td>
Review Comment:
The default value should be 0 I think to preserve the current behavior as this does not affect all kinds of Kubernetes clusters .
many are actually fixed sized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#issuecomment-1516184687
Closing for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mxm commented on a diff in pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #549:
URL: https://github.com/apache/flink-kubernetes-operator/pull/549#discussion_r1143517158
##########
docs/layouts/shortcodes/generated/dynamic_section.html:
##########
@@ -86,6 +86,12 @@
<td>Duration</td>
<td>Interval at which periodic savepoints will be triggered. The triggering schedule is not guaranteed, savepoints will be triggered as part of the regular reconcile loop.</td>
</tr>
+ <tr>
+ <td><h5>kubernetes.operator.rescaling.cluster-cooldown</h5></td>
+ <td style="word-wrap: break-word;">1 min</td>
+ <td>Duration</td>
Review Comment:
I'm ok with not enforcing a default. Fairness is not a big concern because there is an additional per-pipeline stabilization and metric collection phase which will allow other pipelines to be scaled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org