[jira] [Commented] (FLINK-24553) Change buffer debloating default configuration values

["Anton Kalashnikov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435850#comment-17435850 ] 

Anton Kalashnikov commented on FLINK-24553:
-------------------------------------------

I propose the following values:
 * taskmanager.memory.min-segment-size = 256, since it works pretty well when the record size is small and backpressure is high. at the same time, there are no problems in other cases were found.
 * taskmanager.network.memory.buffer-debloat.period = 200, according to the math the value can be stabilized on 70% level for 10 attempts(with default configuration). So since we have debloat target equal to 1000ms it makes sense to set debloat period to 100ms(100 * 10 = 1000ms) but on the other hand, having the stabilization time equal to debloat target time is a little too sensitive to throughput volatiles. So according to experiments, the value 200ms is a good trade-off between the time of reaction and spikes ignoring.
 * taskmanager.network.memory.buffer-debloat.threshold-percentages = 25, logically if considering the default values like the buffer size(32Kb), it looks like the value should be changed too significantly for applying it. (from 32Kb to 16Kb). In my opinion, even if the value has changed from 32Kb to ~20Kb it makes sense to apply this value. Of course, the situation is different for the small value, but perhaps it is even better to change buffer size more frequent when the backpressure is high

 

> Change buffer debloating default configuration values
> -----------------------------------------------------
>
>                 Key: FLINK-24553
>                 URL: https://issues.apache.org/jira/browse/FLINK-24553
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>    Affects Versions: 1.14.0
>            Reporter: Anton Kalashnikov
>            Assignee: Anton Kalashnikov
>            Priority: Major
>
> After the investigation for buffer debloating effectiveness, there are some conclusion:
>  * taskmanager.memory.min-segment-size can be decreased from 1024 to at least 256 because in some cases even 1024 is a too big value and at the same time the low min value is not a problem.
>  * taskmanager.network.memory.buffer-debloat.samples can be decreased from 20 to 10 or taskmanager.network.memory.buffer-debloat.period can be decreased from 500ms to 100ms or 200ms. According to the investigation, the current speed of reaction is too slow so it is better to increase it by changing one of these parameters.
>  * taskmanager.network.memory.buffer-debloat.threshold-percentages can be decreased from 50 to 10 because there are no problems were found when the announcement of buffer size happened more frequently but it actually can positively influent the checkpoint time.
> it is better to change the default value for min-segment-size only after the task https://issues.apache.org/jira/browse/FLINK-24190 will be done.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)