You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2024/02/23 08:21:11 UTC

[PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

1996fanrui opened a new pull request, #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783

   ## What is the purpose of the change
   
   JobVertexScaler#scale has a optimization: Try to adjust the parallelism such that it divides the number of key groups without a remainder =>  data is evenly spread across subtasks.
   
   It's only useful when the upstream shuffle type has keyBy. We should avoid this optimization when the upstream shuffle type doesn't have keyBy.
   
   ## Brief change log
   
   [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy
   
   ## Verifying this change
   
   I'm developing the unit test.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
     - Core observer or reconciler logic that is regularly executed: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501752635


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +228,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue(HASH);

Review Comment:
   Thanks a lot for pointing it out. I forgot that source needs to balance the partition before.
   
   I have updated this PR in the third commit: The source can adjust parallelism for now. Also, I updated parameter names and the comment for `JobVertexScaler#scale` method, for exampe: changing the numKeyGroups to maxParallelism, and changing hasKeyBy to adjustByMaxParallelism.
   
   Because the numKeyGroups and hasKeyBy aren't exactly for source vertex.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502247084


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -245,50 +247,69 @@ private boolean detectIneffectiveScaleUp(
         }
     }
 
+    /**
+     * Compute newParallelism according to currentParallelism.
+     *
+     * @param currentParallelism The current parallelism.
+     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
+     *     it's partition number for kafka source vertex.
+     * @param scaleFactor The scale factor.
+     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
+     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
+     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
+     *     maxParallelism to ensure the keyGroup or partition evenly.
+     */

Review Comment:
   (I'm suggesting to remove the `@param` lines)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#issuecomment-1963892760

   Thanks a lot @mxm for the quick review again, updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501752635


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +228,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue(HASH);

Review Comment:
   Thanks a lot for pointing it out. I forgot it before.
   
   I have updated. The source can adjust parallelism for now. Also, I updated parameter names and the comment for `JobVertexScaler#scale` method, for exampe: changing the numKeyGroups to maxParallelism, and changing hasKeyBy to adjustByMaxParallelism.
   
   Because the numKeyGroups and hasKeyBy aren't exactly for source vertex.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502221272


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -245,50 +247,69 @@ private boolean detectIneffectiveScaleUp(
         }
     }
 
+    /**
+     * Compute newParallelism according to currentParallelism.
+     *
+     * @param currentParallelism The current parallelism.
+     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
+     *     it's partition number for kafka source vertex.
+     * @param scaleFactor The scale factor.
+     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
+     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
+     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
+     *     maxParallelism to ensure the keyGroup or partition evenly.
+     */

Review Comment:
   I prefer adding just an explanation of what this method does. There is no additional information which is not already in the method signature. If we want to clarify, e.g. maxParallelism, we can add a dedicated comment for that.



##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -71,6 +71,7 @@ public JobVertexScaler(AutoScalerEventHandler<KEY, Context> autoScalerEventHandl
     public int computeScaleTargetParallelism(
             Context context,
             JobVertexID vertex,
+            boolean adjustByMaxParallelism,

Review Comment:
   Can we pass-through the ShipStrategy here? That way we decide not based on some boolean but on the relevant data for making the key group adjustment decision. Also makes unit testing more solid.



##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/topology/ShipStrategy.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.topology;
+
+import javax.annotation.Nonnull;
+
+/** The ship strategy between 2 JobVertices. */
+public enum ShipStrategy {

Review Comment:
   Would be nice to reuse Flink's ShipStrategyType if possible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502427965


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -248,25 +250,22 @@ private boolean detectIneffectiveScaleUp(
     }
 
     /**
-     * Compute newParallelism according to currentParallelism.
+     * Computing the newParallelism. In general, newParallelism = currentParallelism * scaleFactor.
+     * But we limit newParallelism between parallelismLowerLimit and min(parallelismUpperLimit,
+     * maxParallelism).
      *
-     * @param currentParallelism The current parallelism.
-     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
-     *     it's partition number for kafka source vertex.
-     * @param scaleFactor The scale factor.
-     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
-     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
-     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
-     *     maxParallelism to ensure the keyGroup or partition evenly.
+     * <p>Also, in order to ensure the data is evenly spread across subtasks, we try to adjust the
+     * parallelism for source and keyed vertex such that it divides the maxParallelism without a
+     * remainder.
      */
     @VisibleForTesting
     protected static int scale(
             int currentParallelism,
+            Map<JobVertexID, ShipStrategy> inputs,

Review Comment:
   This could be reduced to just `Collection<ShipStrategy>`, we don't need the vertex.



##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -248,25 +250,22 @@ private boolean detectIneffectiveScaleUp(
     }
 
     /**
-     * Compute newParallelism according to currentParallelism.
+     * Computing the newParallelism. In general, newParallelism = currentParallelism * scaleFactor.
+     * But we limit newParallelism between parallelismLowerLimit and min(parallelismUpperLimit,
+     * maxParallelism).
      *
-     * @param currentParallelism The current parallelism.
-     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
-     *     it's partition number for kafka source vertex.
-     * @param scaleFactor The scale factor.
-     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
-     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
-     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
-     *     maxParallelism to ensure the keyGroup or partition evenly.
+     * <p>Also, in order to ensure the data is evenly spread across subtasks, we try to adjust the
+     * parallelism for source and keyed vertex such that it divides the maxParallelism without a
+     * remainder.
      */
     @VisibleForTesting
     protected static int scale(
             int currentParallelism,
+            Map<JobVertexID, ShipStrategy> inputs,

Review Comment:
   This could be reduced to just `Collection<ShipStrategy>`, we don't need the vertex id.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1500903357


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +227,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue("HASH");

Review Comment:
   It's a bit strange to check string values like `HASH` can we make this an enum so that it's more explicit in the autoscaler code and easier to handle?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501648975


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +228,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue(HASH);

Review Comment:
   I assume this also means that sources are also not adjusted which can lead to partition imbalance for Kafka topics. We should either keep the adjustments for sources or remove the whole partition number querying/max parallelism logic that we have during collection. 
   
   Overall I think for sources this makes sense and if we want to remove it that should be a separate ticket.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501754751


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -229,13 +229,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
 
-                                final boolean hasKeyBy =
-                                        jobTopology.get(v).getInputs().containsValue(HASH);
+                                var inputs = jobTopology.get(v).getInputs();
+                                var adjustByMaxParallelism =
+                                        inputs.isEmpty() || inputs.containsValue(HASH);
                                 var newParallelism =
                                         jobVertexScaler.computeScaleTargetParallelism(
                                                 context,
                                                 v,
-                                                hasKeyBy,
+                                                adjustByMaxParallelism,
                                                 metrics,

Review Comment:
   Can we please add a single unit test case to cover this core piece of the logic? Just to make sure we don't mess up sources in the future unintentionally  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501819781


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -229,13 +229,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
 
-                                final boolean hasKeyBy =
-                                        jobTopology.get(v).getInputs().containsValue(HASH);
+                                var inputs = jobTopology.get(v).getInputs();
+                                var adjustByMaxParallelism =
+                                        inputs.isEmpty() || inputs.containsValue(HASH);
                                 var newParallelism =
                                         jobVertexScaler.computeScaleTargetParallelism(
                                                 context,
                                                 v,
-                                                hasKeyBy,
+                                                adjustByMaxParallelism,
                                                 metrics,

Review Comment:
   Thank you very much for your review, especially on the weekend. ❤️



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502425862


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/topology/ShipStrategy.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.topology;
+
+import javax.annotation.Nonnull;
+
+/** The ship strategy between 2 JobVertices. */
+public enum ShipStrategy {

Review Comment:
   Sounds good! Thanks for looking into it. Let's keep it as-is for now.



##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -71,6 +71,7 @@ public JobVertexScaler(AutoScalerEventHandler<KEY, Context> autoScalerEventHandl
     public int computeScaleTargetParallelism(
             Context context,
             JobVertexID vertex,
+            boolean adjustByMaxParallelism,

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502428640


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -229,14 +228,11 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
 
-                                var inputs = jobTopology.get(v).getInputs();
-                                var adjustByMaxParallelism =
-                                        inputs.isEmpty() || inputs.containsValue(HASH);
                                 var newParallelism =
                                         jobVertexScaler.computeScaleTargetParallelism(
                                                 context,
                                                 v,
-                                                adjustByMaxParallelism,
+                                                jobTopology.get(v).getInputs(),

Review Comment:
   getInputs().values() would suffice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1500874435


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -280,11 +283,14 @@ protected static int scale(
         // Apply min/max parallelism
         newParallelism = Math.min(Math.max(minParallelism, newParallelism), upperBound);
 
-        // Try to adjust the parallelism such that it divides the number of key groups without a
-        // remainder => state is evenly spread across subtasks
-        for (int p = newParallelism; p <= numKeyGroups / 2 && p <= upperBound; p++) {
-            if (numKeyGroups % p == 0) {
-                return p;
+        // When the shuffle type of vertex data source contains keyBy, we try to adjust the
+        // parallelism such that it divides the number of key groups without a remainder =>
+        // data is evenly spread across subtasks
+        if (hasKeyBy) {

Review Comment:
   Hi, @gyfora @mxm . During I test autoscaler, I found the parallelissm adjustment should not take effect for non-keyed operator.
   
   It's better to include this optimization in 1.8.0. Would you mind helping take a look in your free time? thanks in advanced~
   
   BTW, how about rename  `hasKeyBy` to `isKeyedVertex`?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1500899998


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -280,11 +283,14 @@ protected static int scale(
         // Apply min/max parallelism
         newParallelism = Math.min(Math.max(minParallelism, newParallelism), upperBound);
 
-        // Try to adjust the parallelism such that it divides the number of key groups without a
-        // remainder => state is evenly spread across subtasks
-        for (int p = newParallelism; p <= numKeyGroups / 2 && p <= upperBound; p++) {
-            if (numKeyGroups % p == 0) {
-                return p;
+        // When the shuffle type of vertex data source contains keyBy, we try to adjust the
+        // parallelism such that it divides the number of key groups without a remainder =>
+        // data is evenly spread across subtasks
+        if (hasKeyBy) {

Review Comment:
   Can we turn this `if` inside out and return early if `!hasKeyBy` ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501652946


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +228,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue(HASH);

Review Comment:
   This is related to https://issues.apache.org/jira/browse/FLINK-32119 cc @mxm 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#issuecomment-1962222948

   Hi @gyfora , thank you for the quick review!
   
   I have addressed all you comments. For the string shipStrategy, I extracted a `ShipStrategy` enum to refactor it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502426155


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -245,50 +247,69 @@ private boolean detectIneffectiveScaleUp(
         }
     }
 
+    /**
+     * Compute newParallelism according to currentParallelism.
+     *
+     * @param currentParallelism The current parallelism.
+     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
+     *     it's partition number for kafka source vertex.
+     * @param scaleFactor The scale factor.
+     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
+     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
+     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
+     *     maxParallelism to ensure the keyGroup or partition evenly.
+     */

Review Comment:
   Awesome, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501652946


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -225,10 +228,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                             } else {
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
+
+                                final boolean hasKeyBy =
+                                        jobTopology.get(v).getInputs().containsValue(HASH);

Review Comment:
   This is related to https://issues.apache.org/jira/browse/FLINK-32119



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1501819614


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##########
@@ -229,13 +229,14 @@ Map<JobVertexID, ScalingSummary> computeScalingSummary(
                                 var currentParallelism =
                                         (int) metrics.get(ScalingMetric.PARALLELISM).getCurrent();
 
-                                final boolean hasKeyBy =
-                                        jobTopology.get(v).getInputs().containsValue(HASH);
+                                var inputs = jobTopology.get(v).getInputs();
+                                var adjustByMaxParallelism =
+                                        inputs.isEmpty() || inputs.containsValue(HASH);
                                 var newParallelism =
                                         jobVertexScaler.computeScaleTargetParallelism(
                                                 context,
                                                 v,
-                                                hasKeyBy,
+                                                adjustByMaxParallelism,
                                                 metrics,

Review Comment:
   Good suggestion, added~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502421187


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -245,50 +247,69 @@ private boolean detectIneffectiveScaleUp(
         }
     }
 
+    /**
+     * Compute newParallelism according to currentParallelism.
+     *
+     * @param currentParallelism The current parallelism.
+     * @param maxParallelism The max parallelism for job vertices. It's numKeyGroups by default, and
+     *     it's partition number for kafka source vertex.
+     * @param scaleFactor The scale factor.
+     * @param parallelismLowerLimit The parallelism lower limitation in autoscaler option.
+     * @param parallelismUpperLimit The parallelism upper limitation in autoscaler option.
+     * @param adjustByMaxParallelism True means we need to adjust parallelism according to the
+     *     maxParallelism to ensure the keyGroup or partition evenly.
+     */

Review Comment:
   I updated the comment for this method, please help check again in your free time, thanks~



##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/topology/ShipStrategy.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.topology;
+
+import javax.annotation.Nonnull;
+
+/** The ship strategy between 2 JobVertices. */
+public enum ShipStrategy {

Review Comment:
   I checked the `ShipStrategyType` just now, I found it's different with our requirement. For example, ShipStrategyType has `PARTITION_FORCED_REBALANCE` instead of `REBALANCE`. 
   
   As you said before, current ship-strategy comes from the toString, so IIUC flink repo doesn't have corresponding enum so far. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34504][autoscaler] Avoid the parallelism adjustment when the upstream shuffle type doesn't have keyBy [flink-kubernetes-operator]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #783:
URL: https://github.com/apache/flink-kubernetes-operator/pull/783#discussion_r1502422263


##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##########
@@ -71,6 +71,7 @@ public JobVertexScaler(AutoScalerEventHandler<KEY, Context> autoScalerEventHandl
     public int computeScaleTargetParallelism(
             Context context,
             JobVertexID vertex,
+            boolean adjustByMaxParallelism,

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org