You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Gyula Fora (Jira)" <ji...@apache.org> on 2024/12/20 08:41:00 UTC
[jira] [Comment Edited] (FLINK-36940) Why does autoscaler Math.round the TargetProcessingCapacity
[ https://issues.apache.org/jira/browse/FLINK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907300#comment-17907300 ]
Gyula Fora edited comment on FLINK-36940 at 12/20/24 8:40 AM:
--------------------------------------------------------------
Good question , I don’t remember right now . For the metrics we round to 2/3 decimal digits to improve logging and greatly reduce storage costs
here I guess rounding could be avoided or apply the same 3 decimal point logic
was (Author: gyfora):
Good question , I don’t remember right now . For the metrics we round to 2/3 decimal digits to improve logging and greatly reduce storage costs
here I guess rounding could be avoided
> Why does autoscaler Math.round the TargetProcessingCapacity
> -----------------------------------------------------------
>
> Key: FLINK-36940
> URL: https://issues.apache.org/jira/browse/FLINK-36940
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: Rui Fan
> Priority: Major
>
> AutoScalerUtils#getTargetProcessingCapacity[1] calls
> Math.round(lagCatchupTargetRate + restartCatchupRate + inputTargetAtUtilization);
>
> The return type is double, I don't know why need Math.round here.
>
> I'm writing some end to end tests[2] for autoscaler, and the recommended parallelism is unexpected. After analysis, I found Math.round is the root cause. Following is my test[2] core logic:
> The TARGET_UTILIZATION is 0.8, and BusyTimePerSec is always 800 for source, so we expect the source parallelism won't be changed.
> But the recommended parallelism is changed from 100 to 96.
>
> Note: The processing rate of production jobs is usually very high, so rounding has almost no effect on production jobs.
>
> [1] [https://github.com/apache/flink-kubernetes-operator/blob/091e803a6ae713ebe839742694ab6ca53249c4dd/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/AutoScalerUtils.java#L75]
> [2] [https://github.com/1996fanrui/flink-kubernetes-operator/commit/5420f59a12e924f5f47a7dde8d79f5da9a2b4917#diff-b922ac4031d391fa030b25fba77b453736d43156a0edef5b436fee1d8241295fR158]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)