You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Rui Fan (Jira)" <ji...@apache.org> on 2023/04/06 03:16:00 UTC
[jira] [Commented] (FLINK-31610) Refactoring of LocalBufferPool
[ https://issues.apache.org/jira/browse/FLINK-31610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709173#comment-17709173 ]
Rui Fan commented on FLINK-31610:
---------------------------------
Thanks [~akalash] driving this JIRA, and thanks [~Weijie Guo] [~pnowojski] 's discussion. Sorry for the late response.
{quote}Consider such a scenario, the {{{}CurrentPoolSize = 5{}}}, {{{}numOfRequestedMemorySegments = 7{}}}, {{{}maxOverdraftBuffersPerGate = 2{}}}. If {{{}numberOfRequestedOverdraftMemorySegments = 0{}}}, then 2 buffers can be requested now.
It only happens when poolSize changes.
{quote}
From this case and the definition of the overdraft buffer, it shouldn't be able to request now. General principle: new buffers can be requested when "{_}numOfRequestedMemorySegments + numberOfRequestedOverdraftMemorySegments < poolSize + maxOverdraftBuffersPerGate"{_}.
For the code design, [~Weijie Guo] and me discussed it in this PR[1] before, I didn't find this case at that time. If the case can happen, I think we should convert _{{numberOfRequestedMemorySegments}}_ to {{_numberOfRequestedOverdraftMemorySegments_ when poolSize is decreased.}}
{quote}I can propose getting rid of numberOfRequestedOverdraftMemorySegments and using existing numberOfRequestedMemorySegments instead.
{quote}
Overall, it think it is feasible. The new buffers can be requested when {_}"numberOfRequestedMemorySegments < poolSize + maxOverdraftBuffersPerGate"{_}:
* When _numberOfRequestedMemorySegments <= poolSize,_ all buffers are ordinary buffer
* When _numberOfRequestedMemorySegments > poolSize,_ the `{_}ordinary buffer size = poolSize`{_}, and `{_}the overdraft buffer size = numberOfRequestedMemorySegments - poolSize`{_}
Please correct me if I'm wrong.
[1] https://github.com/apache/flink/pull/22084/files#r1128926904
> Refactoring of LocalBufferPool
> ------------------------------
>
> Key: FLINK-31610
> URL: https://issues.apache.org/jira/browse/FLINK-31610
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.17.0
> Reporter: Anton Kalashnikov
> Priority: Major
>
> FLINK-31293 bug highlighted the issue with the internal mutual consistency of different fields in LocalBufferPool. ex.:
> - `numberOfRequestedOverdraftMemorySegments`
> - `numberOfRequestedMemorySegments`
> - `availableMemorySegment`
> - `currentPoolSize`
> Most of the problem was fixed already(I hope) but it is a good idea to reorganize the code in such a way that all invariants between all fields inside will be clearly determined and difficult to break.
> As one example I can propose getting rid of numberOfRequestedOverdraftMemorySegments and using existing numberOfRequestedMemorySegments instead. That means:
> - the pool will be available when `!availableMemorySegments.isEmpty() && unavailableSubpartitionsCount == 0`
> - we don't request a new `ordinary` buffer when `numberOfRequestedMemorySegments >= currentPoolSize` but we request the overdraft buffer instead
> - `setNumBuffers` should work automatically without any changes
> I think we can come up with a couple of such improvements to simplify the code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)