You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by trionesadam <gi...@git.apache.org> on 2018/04/03 03:40:44 UTC

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

GitHub user trionesadam opened a pull request:

    https://github.com/apache/flink/pull/5802

    [FLINK-9087] [runtime] close the BufferConsumer in RecordWriter.broad…

    ## What is the purpose of the change
    
    BufferConsumer  is Closeable, we'd better close it at last, 
    
    
    ## Brief change log
    
    RecordWriter.broadcastEvent()  is called in StreamTask., RecordWriterOutput ,IterationIntermediateTask and IterationHeadTaskbroadcastEvent, also notice that no one make use of the BufferConsumer returned by broadcastEvent(), so i think the better way to close the return value in RecordWriter. change  the method signature from   BufferConsumer to void, and close the BufferConsumer in the end.
    
    
    ## Verifying this change
    
    This change is a trivial rework / code cleanup without any test coverage.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/trionesadam/flink FLINK-9087-close-broadcastEvent

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5802
    
----
commit a8efee487dd2fcd3bc95732c9676f5ed71d19bf0
Author: triones.deng <tr...@...>
Date:   2018-04-03T02:28:52Z

    [FLINK-9087] [runtime] close the BufferConsumer in RecordWriter.broadcastEvent()

----


---

[GitHub] flink issue #5802: [FLINK-9087] [runtime] change the method signature of Rec...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/5802
  
    merging.


---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5802#discussion_r179453256
  
    --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java ---
    @@ -309,8 +309,6 @@ public void testBroadcastEventBufferReferenceCounting() throws Exception {
     		for (int i = 0; i < queues.length; i++) {
     			assertTrue(parseBuffer(queues[i].remove(), i).isEvent());
     		}
    -
    -		assertTrue(bufferConsumer.isRecycled());
     	}
    --- End diff --
    
    yes, this should not be dropped since this line is the actual test - how about this instead:
    ```
    		// Verify added to all queues
    		assertEquals(1, queues[0].size());
    		assertEquals(1, queues[1].size());
    		
    		// get references to buffer consumers (copies from the original event buffer consumer)
    		BufferConsumer bufferConsumer1 = queues[0].getFirst();
    		BufferConsumer bufferConsumer2 = queues[1].getFirst();
    
    		// process all collected events (recycles the buffer)
    		assertTrue(parseBuffer(queues[0].remove(), 0).isEvent());
    		assertTrue(parseBuffer(queues[1].remove(), 1).isEvent());
    
    		assertTrue(bufferConsumer1.isRecycled());
    		assertTrue(bufferConsumer2.isRecycled());
    ```


---

[GitHub] flink issue #5802: [FLINK-9087] [runtime] change the method signature of Rec...

Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on the issue:

    https://github.com/apache/flink/pull/5802
  
    @trionesadam 👍 


---

[GitHub] flink issue #5802: [FLINK-9087] [runtime] change the method signature of Rec...

Posted by trionesadam <gi...@git.apache.org>.
Github user trionesadam commented on the issue:

    https://github.com/apache/flink/pull/5802
  
    This PR is ready for having another review. @NicoK , @tedyu , Thank you. 
    @tedyu , do we need change the description of this jira?


---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] change the method signature...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5802


---

[GitHub] flink issue #5802: [FLINK-9087] [runtime] change the method signature of Rec...

Posted by trionesadam <gi...@git.apache.org>.
Github user trionesadam commented on the issue:

    https://github.com/apache/flink/pull/5802
  
    @NicoK  thanks a lot


---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5802#discussion_r178707684
  
    --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java ---
    @@ -309,8 +309,6 @@ public void testBroadcastEventBufferReferenceCounting() throws Exception {
     		for (int i = 0; i < queues.length; i++) {
     			assertTrue(parseBuffer(queues[i].remove(), i).isEvent());
     		}
    -
    -		assertTrue(bufferConsumer.isRecycled());
     	}
    --- End diff --
    
    How can you verify your ```bufferConsumer.isRecycled``` logic here if you drop this ?


---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5802#discussion_r179451610
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java ---
    @@ -164,7 +164,7 @@ public BufferConsumer broadcastEvent(AbstractEvent event) throws IOException {
     			if (flushAlways) {
     				flushAll();
     			}
    -			return eventBufferConsumer;
    --- End diff --
    
    You don't need to close the `eventBufferConsumer` since the try-with-resources should already do that. Returning the closed value, however, is kind of strange and only needed in one test. Let's make the method return `void` as you suggested and adapt the test.


---