You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "jrthe42 (via GitHub)" <gi...@apache.org> on 2023/12/13 04:03:30 UTC
[PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]
jrthe42 opened a new pull request, #730:
URL: https://github.com/apache/flink-kubernetes-operator/pull/730
## What is the purpose of the change
This pull request tries to fix a bug, that when the autoscaler decides to scale up/down a standalone application deployment, the taskmanager's replica number does not change accordingly.
## Brief change log
- When calculating the number of tm replicas, give priority to `PipelineOptions.PARALLELISM_OVERRIDES`
## Verifying this change
Unit tests + manual validation
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changes to the `CustomResourceDescriptors`:no)
- Core observer or reconciler logic that is regularly executed: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #730:
URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425156114
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java:
##########
@@ -375,9 +376,24 @@ private int getParallelism() {
* effectiveConfig.get(TaskManagerOptions.NUM_TASK_SLOTS);
}
+ Optional<Integer> maxOverrideParallelism = getMaxParallelismFromOverrideConfig();
+ if (maxOverrideParallelism.isPresent() && maxOverrideParallelism.get() > 0) {
+ return maxOverrideParallelism.get();
+ }
+
return spec.getJob().getParallelism();
}
+ private Optional<Integer> getMaxParallelismFromOverrideConfig() {
Review Comment:
If this method is used to determine the number of task slots, that won't work if slot sharing is disabled or if multiple slot sharing groups are used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]
Posted by "mbalassi (via GitHub)" <gi...@apache.org>.
mbalassi merged PR #730:
URL: https://github.com/apache/flink-kubernetes-operator/pull/730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #730:
URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425156114
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java:
##########
@@ -375,9 +376,24 @@ private int getParallelism() {
* effectiveConfig.get(TaskManagerOptions.NUM_TASK_SLOTS);
}
+ Optional<Integer> maxOverrideParallelism = getMaxParallelismFromOverrideConfig();
+ if (maxOverrideParallelism.isPresent() && maxOverrideParallelism.get() > 0) {
+ return maxOverrideParallelism.get();
+ }
+
return spec.getJob().getParallelism();
}
+ private Optional<Integer> getMaxParallelismFromOverrideConfig() {
Review Comment:
If this method is used to determine the number of required task slots to run the job, that won't work if slot sharing is disabled or if multiple slot sharing groups are used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #730:
URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425170026
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java:
##########
@@ -375,9 +376,24 @@ private int getParallelism() {
* effectiveConfig.get(TaskManagerOptions.NUM_TASK_SLOTS);
}
+ Optional<Integer> maxOverrideParallelism = getMaxParallelismFromOverrideConfig();
+ if (maxOverrideParallelism.isPresent() && maxOverrideParallelism.get() > 0) {
+ return maxOverrideParallelism.get();
+ }
+
return spec.getJob().getParallelism();
}
+ private Optional<Integer> getMaxParallelismFromOverrideConfig() {
Review Comment:
You are completely right @mxm however this is a general limitation of the standalone mode itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org