[jira] [Commented] (FLINK-7394) Manage exclusive buffers in RemoteInputChannel

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179224#comment-16179224 ] 

ASF GitHub Bot commented on FLINK-7394:
---------------------------------------

Github user NicoK commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4499#discussion_r140821090
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java ---
    @@ -183,18 +214,40 @@ void notifySubpartitionConsumed() {
     	}
     
     	/**
    -	 * Releases all received buffers and closes the partition request client.
    +	 * Releases all received and available buffers, closes the partition request client.
     	 */
     	@Override
     	void releaseAllResources() throws IOException {
     		if (isReleased.compareAndSet(false, true)) {
    +
    +			final List<MemorySegment> recyclingSegments = new ArrayList<>();
    +
     			synchronized (receivedBuffers) {
     				Buffer buffer;
     				while ((buffer = receivedBuffers.poll()) != null) {
    -					buffer.recycle();
    +					if (buffer.getRecycler() == this) {
    +						recyclingSegments.add(buffer.getMemorySegment());
    --- End diff --
    
    1) I think, performance is not much of an issue here, as this is only called during take-down of a connection and the overhead of `Buffer#recycle` is actually not that much.
    2) Sorry, but I don't get your concern. Why would you need an extra check when using `buffer.recycle()` instead of `exclusiveRecyclingSegments.add(buffer.getMemorySegment())`? There shouldn't be anything special for the exclusive buffers in this regard compared to ordinary buffers (which is the beauty of the design).
    
    Let me give the example of how `LocalBufferPool` handles this inside `lazyDestroy`: it returns every memory segment (one by one) with `networkBufferPool.recycle()` and, at its end, it is calling `networkBufferPool.destroyBufferPool()` so that the book-keeping inside the `NetworkBufferPool` is updated and buffers may be re-distributed to other `LocalBufferPool` instances.
    We could do this similarly here: recycle one by one, then call some method to update book-keeping and balancing inside `NetworkBufferPool`.


> Manage exclusive buffers in RemoteInputChannel
> ----------------------------------------------
>
>                 Key: FLINK-7394
>                 URL: https://issues.apache.org/jira/browse/FLINK-7394
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Core
>            Reporter: zhijiang
>            Assignee: zhijiang
>             Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control. 
> The basic works are:
> * Exclusive buffers are assigned to {{RemoteInputChannel}} after created by {{SingleInputGate}}.
> * {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage the exclusive buffers itself.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)