[jira] [Commented] (FLINK-26762) Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being blocked

["Anton Kalashnikov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518862#comment-17518862 ] 

Anton Kalashnikov commented on FLINK-26762:
-------------------------------------------

[~fanrui], Thanks for your proposal. It looks valid. As I understand, the main benefit is to wait for an available buffer outside the operator which indeed allows starting checkpoint while there are no available buffers. So it makes sense. I just want to share some of my concerns/notices about this:
* We need to formulate more specifically what is the target case for this improvement. For example, you said about a large record but in fact, I don't think it is valid since we already reset to unavailability when at least one buffer is available yet. This means that this large record should be greater than the size of the segment but in this case, we strongly recommend increasing the segment size to avoid such a situation. at the same time `flatmap` is valid case I think, what else? window operator?
* I am also a little concerned about the new configuration. Since we already have a lot of buffer configurations(exclusive/floating buffers number and size, max buffers per channel) which are not so easy to setup correctly. Unfortunately, I don't see any other good solution right now. But I just want emphasize that we should be careful with new configuration since it is not always easy for understanding.
* It is also a good idea to understand how helpful is this improvement. So perhaps, it makes sense to have a benchmark for it or use some existing benchmark. But any case, we should somehow understand that it works as we expected.
* I don't think that it will predictably work with buffer debloating. Maybe it is a not so big problem but anyway we should take it into account.

One more idea for future improvement, we can adjust the number of overdraft buffers or even overdraft buffer size automatically as we do it for buffer debloating now. For example, we can calculate median/average  operator output size and always keep the same size of memory available. Perhaps in this case we don't even need configuration for exact number of overdraft buffers (but we highly likely need another configuration for managing this algorithm).

P. S. [~fanrui], do you want to work on this task yourself?

> Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being blocked
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-26762
>                 URL: https://issues.apache.org/jira/browse/FLINK-26762
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing, Runtime / Network
>    Affects Versions: 1.13.0, 1.14.0, 1.15.0
>            Reporter: fanrui
>            Priority: Major
>             Fix For: 1.16.0
>
>
> In some past JIRAs of Unaligned Checkpoint, the community has added the  recordWriter.isAvaliable() to reduce block for single record write. But for large record, flatmap or broadcast watermark, they may need more buffer.
> Can we add the overdraft buffer in BufferPool to reduce unaligned checkpoint being blocked? 
> h2. Overdraft Buffer mechanism
> Add the configuration of 'taskmanager.network.memory.overdraft-buffers-per-gate=5'. 
> When requestMemory is called and the bufferPool is insufficient, the bufferPool will allow the Task to overdraw up to 5 MemorySegments. And bufferPool will be unavailable until all overdrawn buffers are consumed by downstream tasks. Then the task will wait for bufferPool being available.
> From the above, we have the following benefits:
>  * For scenarios that require multiple buffers, the Task releases the Checkpoint lock, so the Unaligned Checkpoint can be completed quickly.
>  * We can control the memory usage to prevent memory leak.
>  * It just needs a litter memory, and can improve the stability of the Task under back pressure.
>  * Users can increase the overdraft-buffers to adapt the scenarios that require more buffers.
>  
> Masters, please correct me if I'm wrong, thanks a lot.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)