You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/09/30 10:48:02 UTC

[GitHub] [flink] zhijiangW opened a new pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

zhijiangW opened a new pull request #13523:
URL: https://github.com/apache/flink/pull/13523


   ## What is the purpose of the change
   
   
   ## Brief change log
   
   
   
   ## Verifying this change
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
     - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5bcd004fd5105229faaa61c205ee7c596298f8b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983) 
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 84e51e22e60e09d1263dcc07495dc2dadd2f105e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 975212a86b00dd07f3028e81760f8fe15b29b926 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605) 
   * b789480d93ec0006ff7c1441943611d3a1032a24 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r514180270



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
##########
@@ -238,6 +248,12 @@ public void setup() throws IOException {
 		checkState(this.bufferPool == null, "Bug in input gate setup logic: Already registered buffer pool.");
 		setupChannels();
 
+		//TODO we can further judge the condition whether the type is file-based or mmap-based
+		if (consumedPartitionType.isBlocking()) {

Review comment:
       I thought of another issue to use `Supplier<MemorySegment>` way, that means for every call of `LocalInputChannel#getNextBuffer` we need to allocate a new segment for it (no matter with pooled or unpooled segment). My previous way tried to allocate only one segment for always reuse in all the ``LocalInputChannel#getNextBuffer` calls.
   
   But I am fine with the proposal of using unpooled segment here, because it indeed simplify some aspects as you mentioned above.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27d751d0ef8d786c233bb4c16950a81f6b410e70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498) 
   * 84e51e22e60e09d1263dcc07495dc2dadd2f105e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 84e51e22e60e09d1263dcc07495dc2dadd2f105e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520) 
   * 975212a86b00dd07f3028e81760f8fe15b29b926 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * c012a3b687c0ceab428913a5e6cea6f9d3948932 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655) 
   * 33a3c9cec53624573cc00a6e5e42f0d921b1e231 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-711483920


   @flinkbot run azure


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r516206985



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -371,6 +368,22 @@ ByteBuf write(ByteBufAllocator allocator) throws IOException {
 			}
 		}
 
+		private ByteBuf fillHeader(ByteBufAllocator allocator) {
+			// in order to forward the buffer to netty, it needs an allocator set
+			buffer.setAllocator(allocator);

Review comment:
       It is a bit confusing that this happens in a method called `fillHeader()`. I would suggest to leave this out of this method.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -491,6 +505,7 @@ static ErrorResponse readFrom(ByteBuf buffer) throws Exception {
 	static class PartitionRequest extends NettyMessage {
 
 		private static final byte ID = 2;
+		private static final int LENGTH = 20 + 40 + 4 + 16 + 4;

Review comment:
       NIT: This is an unrelated change.
   
   It also does not necessarily become easier this way. Before, the length computation was inline, and one could see at a glance where the numbers came from (lengths, IDs, etc.) because the contents was written in the lines after. Now, this sum of numbers stands isolated and one does not see easily where the number 4, 20, 16, etc. come from and what they mean.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {

Review comment:
       Can we rename this to `BoundedBlockingSubpartitionDirectTransferReader`? Then the naming scheme is similar to `BoundedBlockingSubpartitionReader`.
   
   It is a bit confusing when some two different implementation on the same level use different naming patterns.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/FileRegionBuffer.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.buffer;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBuf;
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufAllocator;
+import org.apache.flink.shaded.netty4.io.netty.channel.DefaultFileRegion;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Wrapper for {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} instance.

Review comment:
       Would be good to explain a bit more why this class is like it is, and how it works:
   
     - This class implements `Buffer` so we can move it can be used like another `Buffer`, although it behaves "read-only" style, so many methods throw UnsupportedOperationException.
   
     - This extends from Netty's `DefaultFileRegion` so we don't need to do any special handling. Netty will internally treat this differently than the `Buffer` that implements `ByteBuf`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -552,6 +556,7 @@ public String toString() {
 	static class TaskEventRequest extends NettyMessage {
 
 		private static final byte ID = 3;
+		private static final int LENGTH = 4 + 20 + 40 + 16;

Review comment:
       See comment above.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -691,23 +672,12 @@ static CloseRequest readFrom(@SuppressWarnings("unused") ByteBuf buffer) throws
 		}
 
 		@Override
-		ByteBuf write(ByteBufAllocator allocator) throws IOException {
-			ByteBuf result = null;
-
-			try {
-				result = allocateBuffer(allocator, ID, 4 + 16);
-				result.writeInt(credit);
-				receiverId.writeTo(result);
-
-				return result;
-			}
-			catch (Throwable t) {
-				if (result != null) {
-					result.release();
-				}
-
-				throw new IOException(t);
-			}
+		void write(ChannelOutboundInvoker out, ChannelPromise promise, ByteBufAllocator allocator) throws IOException {
+			Consumer<ByteBuf> consumer = (bb) -> {

Review comment:
       Do you think we should avoid introducing another virtual/interface method call here, and another object instantiation for the lambda argument capture, or is that not important at this point?
   
   Maybe for the most common messages that are on the performance path (BufferResponse, AddCredit). BufferResponse is already avoiding this, maybe AddCredit can as well.
   
   If we want to avoid it, we could also go with this pattern:
   
   ```java
   final ByteBuf buffer = allocateBuffer(allocator, ID, length);
   try {
       // same code as in consumer
       buffer.writeInt(....);
       ...
       out.write(result, promise);
   } catch (Throwable t) {
       handleException(buffer, t);
   }
   ```
   That way we only have the virtual call in the exception case, and no extra object. And it looks minimally more verbose/duplicate.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferReaderWriterUtil.java
##########
@@ -187,6 +205,17 @@ static ByteBuffer allocatedHeaderBuffer() {
 		return new ByteBuffer[] { allocatedHeaderBuffer(), null };
 	}
 
+	public static ByteBuffer tryReadByteBuffer(FileChannel channel, int size) {
+		ByteBuffer bb = ByteBuffer.allocate(size);
+		try {
+			tryReadByteBuffer(channel, bb);
+			bb.flip();
+		} catch (IOException ex) {
+			ExceptionUtils.rethrow(ex);

Review comment:
       This does not feel quite right. A IO-related utility method that just turns an IOException into an unchecked exception.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/FileRegionBuffer.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.buffer;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBuf;
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufAllocator;
+import org.apache.flink.shaded.netty4.io.netty.channel.DefaultFileRegion;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Wrapper for {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} instance.
+ *
+ * <p>The file region can be sent via network channel which supports zero-copy file transfer,
+ * and it can also be read into memory segment locally via {@link #readInto(MemorySegment)}.
+ */
+public class FileRegionBuffer extends DefaultFileRegion implements Buffer {
+
+	/** The number of bytes to be read/transferred from this file region. */
+	private final int bufferSize;
+
+	private final FileChannel fileChannel;
+
+	/** The {@link DataType} this buffer represents. */
+	private final DataType dataType;
+
+	/** Whether the buffer is compressed or not. */
+	private final boolean isCompressed;
+
+	public FileRegionBuffer(
+			FileChannel fileChannel,
+			int bufferSize,
+			DataType dataType,
+			boolean isCompressed) throws IOException {
+
+		super(fileChannel, fileChannel.position(), bufferSize);
+
+		this.fileChannel = checkNotNull(fileChannel);
+		this.bufferSize = bufferSize;
+		this.dataType = checkNotNull(dataType);
+		this.isCompressed = isCompressed;
+	}
+
+	// ------------------------------------------------------------------------
+	// Buffer override methods
+	// ------------------------------------------------------------------------
+
+	@Override
+	public boolean isBuffer() {
+		return dataType.isBuffer();
+	}
+
+	@Override
+	public MemorySegment getMemorySegment() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getMemorySegmentOffset() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public ReadOnlySlicedNetworkBuffer readOnlySlice() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public ReadOnlySlicedNetworkBuffer readOnlySlice(int index, int length) {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getMaxCapacity() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getReaderIndex() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void setReaderIndex(int readerIndex) throws IndexOutOfBoundsException {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	/**
+	 * This method is only implemented for tests at the moment.
+	 */
+	@Override
+	public ByteBuffer getNioBufferReadable() {
+		return BufferReaderWriterUtil.tryReadByteBuffer(fileChannel, bufferSize);

Review comment:
       This method here is a little concerning, because if used wrong it voids the whole implementation. It reads into a newly allocated buffer.
   
   I assume this is mainly needed for events?
   
   One alternative here could be that the reading of events from the file channel produces a regular `NetworkBuffer`. Then this method here would not be needed.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {
+
+	/** The result subpartition that we read. */
+	private final BoundedBlockingSubpartition parent;
+
+	/** The reader/decoder to the file region with the data we currently read from. */
+	private final BoundedData.Reader dataReader;
+
+	/** The remaining number of data buffers (not events) in the result. */
+	private int numDataBuffers;
+
+	/** The remaining number of data buffers and events in the result. */
+	private int numDataAndEventBuffers;
+
+	/** Flag whether this reader is released. */
+	private boolean isReleased;
+
+	private int sequenceNumber;
+
+	BoundedBlockingSubpartitionView(
+		BoundedBlockingSubpartition parent,
+		Path filePath,
+		int numDataBuffers,
+		int numDataAndEventBuffers) throws IOException {
+
+		this.parent = checkNotNull(parent);
+
+		checkNotNull(filePath);
+		this.dataReader = new FileRegionReader(filePath);
+
+		checkArgument(numDataBuffers >= 0);
+		this.numDataBuffers = numDataBuffers;
+
+		checkArgument(numDataAndEventBuffers >= 0);
+		this.numDataAndEventBuffers = numDataAndEventBuffers;
+	}
+
+	@Nullable
+	@Override
+	public BufferAndBacklog getNextBuffer() throws IOException {
+		if (isReleased) {
+			return null;
+		}
+
+		Buffer current = dataReader.nextBuffer();
+		if (current == null) {
+			// as per contract, we must return null when the reader is empty,
+			// but also in case the reader is disposed (rather than throwing an exception)
+			return null;
+		}
+
+		updateStatistics(current);
+
+		// We simply assume all the data are non-events for batch jobs to avoid pre-fetching the next header
+		Buffer.DataType nextDataType = numDataAndEventBuffers > 0 ? Buffer.DataType.DATA_BUFFER : Buffer.DataType.NONE;
+		return BufferAndBacklog.fromBufferAndLookahead(current, nextDataType, numDataBuffers, sequenceNumber++);
+	}
+
+	private void updateStatistics(Buffer buffer) {
+		if (buffer.isBuffer()) {
+			numDataBuffers--;
+		}
+		numDataAndEventBuffers--;
+	}
+
+	@Override
+	public boolean isAvailable(int numCreditsAvailable) {
+		// We simply assume there are no events except EndOfPartitionEvent for bath jobs,
+		// then it has no essential effect to ignore the judgement of next event buffer.
+		return numCreditsAvailable > 0 && numDataAndEventBuffers > 0;
+	}
+
+	@Override
+	public void releaseAllResources() throws IOException {
+		// it is not a problem if this method executes multiple times
+		isReleased = true;
+
+		IOUtils.closeQuietly(dataReader);
+
+		// Notify the parent that this one is released. This allows the parent to
+		// eventually release all resources (when all readers are done and the
+		// parent is disposed).
+		parent.releaseReaderReference(this);
+	}
+
+	@Override
+	public boolean isReleased() {
+		return isReleased;
+	}
+
+	@Override
+	public Throwable getFailureCause() {
+		// we can never throw an error after this was created
+		return null;
+	}
+
+	@Override
+	public int unsynchronizedGetNumberOfQueuedBuffers() {
+		return parent.unsynchronizedGetNumberOfQueuedBuffers();
+	}
+
+	@Override
+	public void notifyDataAvailable() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void resumeConsumption() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public String toString() {
+		return String.format("Blocking Subpartition Reader: ID=%s, index=%d",
+			parent.parent.getPartitionId(),
+			parent.getSubPartitionIndex());
+	}
+
+	/**
+	 * The reader to read from {@link BoundedBlockingSubpartition} and return the wrapped
+	 * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} based buffer.
+	 */
+	static final class FileRegionReader implements BoundedData.Reader {
+
+		private final FileChannel fileChannel;
+
+		private final ByteBuffer headerBuffer;
+
+		private long lastPosition = -1L;
+
+		private int lastBufferSize;
+
+		FileRegionReader(Path filePath) throws IOException {
+			this.fileChannel = FileChannel.open(filePath, StandardOpenOption.READ);
+			this.headerBuffer = BufferReaderWriterUtil.allocatedHeaderBuffer();
+		}
+
+		@Nullable
+		@Override
+		public Buffer nextBuffer() throws IOException {
+			// As the file region transfer via network channel will not modify the underlining file position,

Review comment:
       Maybe we can optimize this method a bit.
   
   If you inline the code from BufferReaderWriterUtil.readFromByteChannel() here, then you can avoid having the logic tracking `lastPosition` etc, and simply advance the position always here.
   
   It should also help to avoid some calls to `fileChannel.position()`, which are not cheap.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartition.java
##########
@@ -74,7 +74,10 @@
 
 	/** All created and not yet released readers. */
 	@GuardedBy("lock")
-	private final Set<BoundedBlockingSubpartitionReader> readers;
+	private final Set<ResultSubpartitionView> readers;
+
+	/** Flag whether SSL is enabled for network transfer with file type. */
+	private final boolean fileWithoutSSLEnabled;

Review comment:
       How about renaming this to `useDirectFileTransfer` ?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {
+
+	/** The result subpartition that we read. */
+	private final BoundedBlockingSubpartition parent;
+
+	/** The reader/decoder to the file region with the data we currently read from. */
+	private final BoundedData.Reader dataReader;
+
+	/** The remaining number of data buffers (not events) in the result. */
+	private int numDataBuffers;
+
+	/** The remaining number of data buffers and events in the result. */
+	private int numDataAndEventBuffers;
+
+	/** Flag whether this reader is released. */
+	private boolean isReleased;
+
+	private int sequenceNumber;
+
+	BoundedBlockingSubpartitionView(
+		BoundedBlockingSubpartition parent,
+		Path filePath,
+		int numDataBuffers,
+		int numDataAndEventBuffers) throws IOException {
+
+		this.parent = checkNotNull(parent);
+
+		checkNotNull(filePath);
+		this.dataReader = new FileRegionReader(filePath);
+
+		checkArgument(numDataBuffers >= 0);
+		this.numDataBuffers = numDataBuffers;
+
+		checkArgument(numDataAndEventBuffers >= 0);
+		this.numDataAndEventBuffers = numDataAndEventBuffers;
+	}
+
+	@Nullable
+	@Override
+	public BufferAndBacklog getNextBuffer() throws IOException {
+		if (isReleased) {
+			return null;
+		}
+
+		Buffer current = dataReader.nextBuffer();
+		if (current == null) {
+			// as per contract, we must return null when the reader is empty,
+			// but also in case the reader is disposed (rather than throwing an exception)
+			return null;
+		}
+
+		updateStatistics(current);
+
+		// We simply assume all the data are non-events for batch jobs to avoid pre-fetching the next header
+		Buffer.DataType nextDataType = numDataAndEventBuffers > 0 ? Buffer.DataType.DATA_BUFFER : Buffer.DataType.NONE;
+		return BufferAndBacklog.fromBufferAndLookahead(current, nextDataType, numDataBuffers, sequenceNumber++);
+	}
+
+	private void updateStatistics(Buffer buffer) {
+		if (buffer.isBuffer()) {
+			numDataBuffers--;
+		}
+		numDataAndEventBuffers--;
+	}
+
+	@Override
+	public boolean isAvailable(int numCreditsAvailable) {
+		// We simply assume there are no events except EndOfPartitionEvent for bath jobs,
+		// then it has no essential effect to ignore the judgement of next event buffer.
+		return numCreditsAvailable > 0 && numDataAndEventBuffers > 0;
+	}
+
+	@Override
+	public void releaseAllResources() throws IOException {
+		// it is not a problem if this method executes multiple times
+		isReleased = true;
+
+		IOUtils.closeQuietly(dataReader);
+
+		// Notify the parent that this one is released. This allows the parent to
+		// eventually release all resources (when all readers are done and the
+		// parent is disposed).
+		parent.releaseReaderReference(this);
+	}
+
+	@Override
+	public boolean isReleased() {
+		return isReleased;
+	}
+
+	@Override
+	public Throwable getFailureCause() {
+		// we can never throw an error after this was created
+		return null;
+	}
+
+	@Override
+	public int unsynchronizedGetNumberOfQueuedBuffers() {
+		return parent.unsynchronizedGetNumberOfQueuedBuffers();
+	}
+
+	@Override
+	public void notifyDataAvailable() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void resumeConsumption() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public String toString() {
+		return String.format("Blocking Subpartition Reader: ID=%s, index=%d",
+			parent.parent.getPartitionId(),
+			parent.getSubPartitionIndex());
+	}
+
+	/**
+	 * The reader to read from {@link BoundedBlockingSubpartition} and return the wrapped
+	 * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} based buffer.
+	 */
+	static final class FileRegionReader implements BoundedData.Reader {
+
+		private final FileChannel fileChannel;
+
+		private final ByteBuffer headerBuffer;
+
+		private long lastPosition = -1L;
+
+		private int lastBufferSize;
+
+		FileRegionReader(Path filePath) throws IOException {
+			this.fileChannel = FileChannel.open(filePath, StandardOpenOption.READ);
+			this.headerBuffer = BufferReaderWriterUtil.allocatedHeaderBuffer();
+		}
+
+		@Nullable
+		@Override
+		public Buffer nextBuffer() throws IOException {
+			// As the file region transfer via network channel will not modify the underlining file position,

Review comment:
       I think this is very fragile, using the `lastPosition` to distinguish the uses.
   
   Can we say that this part here (creating regions from the file) always advances the position, and the actual reading part never advances the position? For that, the `FileRegionBuffer#readInto(...)` would also need to take the offset/length fields stored in the region to handle this.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {
+
+	/** The result subpartition that we read. */
+	private final BoundedBlockingSubpartition parent;
+
+	/** The reader/decoder to the file region with the data we currently read from. */
+	private final BoundedData.Reader dataReader;
+
+	/** The remaining number of data buffers (not events) in the result. */
+	private int numDataBuffers;
+
+	/** The remaining number of data buffers and events in the result. */
+	private int numDataAndEventBuffers;
+
+	/** Flag whether this reader is released. */
+	private boolean isReleased;
+
+	private int sequenceNumber;
+
+	BoundedBlockingSubpartitionView(
+		BoundedBlockingSubpartition parent,
+		Path filePath,
+		int numDataBuffers,
+		int numDataAndEventBuffers) throws IOException {
+
+		this.parent = checkNotNull(parent);
+
+		checkNotNull(filePath);
+		this.dataReader = new FileRegionReader(filePath);
+
+		checkArgument(numDataBuffers >= 0);
+		this.numDataBuffers = numDataBuffers;
+
+		checkArgument(numDataAndEventBuffers >= 0);
+		this.numDataAndEventBuffers = numDataAndEventBuffers;
+	}
+
+	@Nullable
+	@Override
+	public BufferAndBacklog getNextBuffer() throws IOException {
+		if (isReleased) {
+			return null;
+		}
+
+		Buffer current = dataReader.nextBuffer();
+		if (current == null) {
+			// as per contract, we must return null when the reader is empty,
+			// but also in case the reader is disposed (rather than throwing an exception)
+			return null;
+		}
+
+		updateStatistics(current);
+
+		// We simply assume all the data are non-events for batch jobs to avoid pre-fetching the next header
+		Buffer.DataType nextDataType = numDataAndEventBuffers > 0 ? Buffer.DataType.DATA_BUFFER : Buffer.DataType.NONE;
+		return BufferAndBacklog.fromBufferAndLookahead(current, nextDataType, numDataBuffers, sequenceNumber++);
+	}
+
+	private void updateStatistics(Buffer buffer) {
+		if (buffer.isBuffer()) {
+			numDataBuffers--;
+		}
+		numDataAndEventBuffers--;
+	}
+
+	@Override
+	public boolean isAvailable(int numCreditsAvailable) {
+		// We simply assume there are no events except EndOfPartitionEvent for bath jobs,
+		// then it has no essential effect to ignore the judgement of next event buffer.
+		return numCreditsAvailable > 0 && numDataAndEventBuffers > 0;
+	}
+
+	@Override
+	public void releaseAllResources() throws IOException {
+		// it is not a problem if this method executes multiple times
+		isReleased = true;
+
+		IOUtils.closeQuietly(dataReader);
+
+		// Notify the parent that this one is released. This allows the parent to
+		// eventually release all resources (when all readers are done and the
+		// parent is disposed).
+		parent.releaseReaderReference(this);
+	}
+
+	@Override
+	public boolean isReleased() {
+		return isReleased;
+	}
+
+	@Override
+	public Throwable getFailureCause() {
+		// we can never throw an error after this was created
+		return null;
+	}
+
+	@Override
+	public int unsynchronizedGetNumberOfQueuedBuffers() {
+		return parent.unsynchronizedGetNumberOfQueuedBuffers();
+	}
+
+	@Override
+	public void notifyDataAvailable() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void resumeConsumption() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public String toString() {
+		return String.format("Blocking Subpartition Reader: ID=%s, index=%d",
+			parent.parent.getPartitionId(),
+			parent.getSubPartitionIndex());
+	}
+
+	/**
+	 * The reader to read from {@link BoundedBlockingSubpartition} and return the wrapped
+	 * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} based buffer.
+	 */
+	static final class FileRegionReader implements BoundedData.Reader {
+
+		private final FileChannel fileChannel;
+
+		private final ByteBuffer headerBuffer;
+
+		private long lastPosition = -1L;
+
+		private int lastBufferSize;
+
+		FileRegionReader(Path filePath) throws IOException {
+			this.fileChannel = FileChannel.open(filePath, StandardOpenOption.READ);
+			this.headerBuffer = BufferReaderWriterUtil.allocatedHeaderBuffer();
+		}
+
+		@Nullable
+		@Override
+		public Buffer nextBuffer() throws IOException {
+			// As the file region transfer via network channel will not modify the underlining file position,

Review comment:
       If you want, I can try and create a suggestion for this change.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3461a87abdb961092e464030700a29964833de7a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125) 
   * a4198d83d630c6efa49d0153142e201426bc6507 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen closed pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen closed pull request #13523:
URL: https://github.com/apache/flink/pull/13523


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW removed a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW removed a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-708994253


   @flinkbot run azure


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3461a87abdb961092e464030700a29964833de7a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27d751d0ef8d786c233bb4c16950a81f6b410e70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-717133357


   Thanks, @zhijiangW - some thoughts on your suggestions here:
   
   > Introduce another FileRegionResponse which extends DefaultFileRegion, because we need to override the #deallocate method not to close file channel for every call.
   
   Do we need to `FileRegionResponse` to extend `DefaultFileRegion`, or can the `FileRegionResponse` contain a `DefaultFileRegion` object? Given the complex interface of `FileRegion` in Netty (and how it may change between versions) might be good to avoid inheriting from it.
   
   You are right about `buffer#recycleBuffer`, `buffer#setAllocator` and `BufferResponse#getBuffer` - we need to handle this differently for buffers and regions.
   
   > Handling SSL
   
   I think SSL is quite important, many setups need it for data security compliance reasons. So we definitely need to support it.
   For the time being it is probably okay to say that the optimization from this PR here is not active when SSL is active, that we keep using the original version with the higher memory use when SSL is active. Then we see how the sort-based shuffle develops, maybe we need to encourage people to use that one when SSL is active.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-722432311


   Rebased and manually merged in 4869d3593d5b7fcff36abbe71a3e1a46b7a64998


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-713465353


   I supplemented a new micro-benchmark case for verifying the change via [PR](https://github.com/apache/flink-benchmarks/pull/4), and the results as below seem no obvious change.
   
   After change:                                                                                                    
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7009.393 ± 251.786  ops/ms
   
   Before change:                                                                                                       
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7109.649 ± 197.832  ops/ms


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) 
   * b5bcd004fd5105229faaa61c205ee7c596298f8b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r512269411



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionData.java
##########
@@ -45,6 +51,14 @@
 
 	public abstract boolean isBuffer();
 
+	/**
+	 * Returns the buffer-format partition data with the provided memory segment or not.
+	 *
+	 * @param segment it might be needed to read the partition data into.
+	 * @return the buffer represents the partition data.
+	 */
+	public abstract Buffer getBuffer(@Nullable MemorySegment segment) throws IOException;

Review comment:
       I would change this to `Supplier<MemorySegment>` so that the code lazily fetches (and allocates) the memory segment only when needed. See also next comment about lazy creation.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferReaderWriterUtil.java
##########
@@ -181,7 +181,7 @@ private static boolean tryReadByteBuffer(FileChannel channel, ByteBuffer b) thro
 		}
 	}
 
-	private static void readByteBufferFully(FileChannel channel, ByteBuffer b) throws IOException {
+	public static void readByteBufferFully(FileChannel channel, ByteBuffer b) throws IOException {

Review comment:
       The remaining class uses package-private visibility, would suggest to use that here as well for consistency.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
##########
@@ -238,6 +248,12 @@ public void setup() throws IOException {
 		checkState(this.bufferPool == null, "Bug in input gate setup logic: Already registered buffer pool.");
 		setupChannels();
 
+		//TODO we can further judge the condition whether the type is file-based or mmap-based
+		if (consumedPartitionType.isBlocking()) {

Review comment:
       I would simply make this a simple lazy-created MemorySegement, without any logic trying to determine what the shuffle type is. If this is fetched only if needed (see using `Supplier<MemorySegment>` above) then that will be good enough.
   
   I think it would also be fine if this was an unpooled segment, not obtained from the `memorySegmentProvider` but just created on first access. Would make things simpler, you don't have to worry about recycling, races between creation and early disposal, etc.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "triggerType" : "PUSH"
     }, {
       "hash" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8667",
       "triggerID" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * c012a3b687c0ceab428913a5e6cea6f9d3948932 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655) 
   * 33a3c9cec53624573cc00a6e5e42f0d921b1e231 UNKNOWN
   * 31de3dfae7e14c5c262d8326611a1dd29ea9b834 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8667) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a4198d83d630c6efa49d0153142e201426bc6507 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409) 
   * 4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701314768


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 3461a87abdb961092e464030700a29964833de7a (Wed Sep 30 10:51:00 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
    * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15981).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27d751d0ef8d786c233bb4c16950a81f6b410e70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498) 
   * 84e51e22e60e09d1263dcc07495dc2dadd2f105e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3461a87abdb961092e464030700a29964833de7a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a4198d83d630c6efa49d0153142e201426bc6507 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409) 
   * 4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-713465353


   I supplemented a new micro-benchmark case for verifying the change via [PR](https://github.com/apache/flink-benchmarks/pull/4), and the results as below seem no obvious change.
   
   Before change:                                                                                                    
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7009.393 ± 251.786  ops/ms
   
   After change:                                                                                                       
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7109.649 ± 197.832  ops/ms


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) 
   * b5bcd004fd5105229faaa61c205ee7c596298f8b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r512529671



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
##########
@@ -238,6 +248,12 @@ public void setup() throws IOException {
 		checkState(this.bufferPool == null, "Bug in input gate setup logic: Already registered buffer pool.");
 		setupChannels();
 
+		//TODO we can further judge the condition whether the type is file-based or mmap-based
+		if (consumedPartitionType.isBlocking()) {

Review comment:
       Agree, it seems better to make it lazy-created and unpooled segment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-721743899


   Thanks! I can address these remaining points and merge this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * c012a3b687c0ceab428913a5e6cea6f9d3948932 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5bcd004fd5105229faaa61c205ee7c596298f8b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-707562596


   FYI, the last four commits separation is only for easy review, will be squashed finally before merging.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 975212a86b00dd07f3028e81760f8fe15b29b926 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605) 
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-713465353


   I supplemented a new micro-benchmark case for verifying the change via [PR](https://github.com/apache/flink-benchmarks/pull/4), and the results as below seem no obvious change.
   
   Before change:                                                                                                    Mode  Cnt     Score     Error   Units
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7009.393 ± 251.786  ops/ms
   
   After change:                                                                                                       Mode  Cnt     Score     Error   Units
   RemoteBlockingPartitionThroughputBenchmark.remoteBlockingPartition  thrpt   30  7109.649 ± 197.832  ops/ms


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-719465267


   @StephanEwen, I have updated the codes for addressing above comments. 
   
   Regards the tests, I think the existing `BoundedBlockingSubpartitionTest` and `BoundedBlockingSubpartitionWriteReadTest` already covered the local read part. 
   I will further consider how to cover the remote network read by existing tests after you approve the current implementation way. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 84e51e22e60e09d1263dcc07495dc2dadd2f105e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520) 
   * 975212a86b00dd07f3028e81760f8fe15b29b926 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b789480d93ec0006ff7c1441943611d3a1032a24 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-708994253


   @flinkbot run azure


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r503059525



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionData.java
##########
@@ -45,6 +51,14 @@
 
 	public abstract boolean isBuffer();
 
+	/**
+	 * Returns the buffer-format partition data with the provided memory segment or not.
+	 *
+	 * @param segment it might be needed to read the partition data into.
+	 * @return the buffer represents the partition data.
+	 */
+	public abstract Buffer getBuffer(@Nullable MemorySegment segment) throws IOException;

Review comment:
       TBH, I am a bit torn and not satisfied with introducing this method, since the argument `segment` seem not very elegant from the caller usage. But I did not think of a better option for resolving it now. :(




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "triggerType" : "PUSH"
     }, {
       "hash" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8667",
       "triggerID" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a15bc75ce36230e77cc96e51ac9e7709be0a4f05",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8752",
       "triggerID" : "a15bc75ce36230e77cc96e51ac9e7709be0a4f05",
       "triggerType" : "PUSH"
     }, {
       "hash" : "decff044e3e02fba711f5b3120cc5762dffdc8ee",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8763",
       "triggerID" : "decff044e3e02fba711f5b3120cc5762dffdc8ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a77c723967fd9d34658bb87fc0c1609538a6eea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8865",
       "triggerID" : "9a77c723967fd9d34658bb87fc0c1609538a6eea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a62e9acbc8293f2d3425f29080bfa1771d7767bf",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8882",
       "triggerID" : "a62e9acbc8293f2d3425f29080bfa1771d7767bf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * 33a3c9cec53624573cc00a6e5e42f0d921b1e231 UNKNOWN
   * a62e9acbc8293f2d3425f29080bfa1771d7767bf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8882) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b5bcd004fd5105229faaa61c205ee7c596298f8b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983) 
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * c012a3b687c0ceab428913a5e6cea6f9d3948932 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a4198d83d630c6efa49d0153142e201426bc6507 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7647",
       "triggerID" : "708994253",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7824",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b789480d93ec0006ff7c1441943611d3a1032a24",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7664",
       "triggerID" : "711483920",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7983",
       "triggerID" : "b5bcd004fd5105229faaa61c205ee7c596298f8b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "81b40b201a250bc3c738b30d3d820657b9f749ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655",
       "triggerID" : "c012a3b687c0ceab428913a5e6cea6f9d3948932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "33a3c9cec53624573cc00a6e5e42f0d921b1e231",
       "triggerType" : "PUSH"
     }, {
       "hash" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "31de3dfae7e14c5c262d8326611a1dd29ea9b834",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 81b40b201a250bc3c738b30d3d820657b9f749ee UNKNOWN
   * c012a3b687c0ceab428913a5e6cea6f9d3948932 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8655) 
   * 33a3c9cec53624573cc00a6e5e42f0d921b1e231 UNKNOWN
   * 31de3dfae7e14c5c262d8326611a1dd29ea9b834 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW removed a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW removed a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-711483920


   @flinkbot run azure


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a4198d83d630c6efa49d0153142e201426bc6507 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409) 
   * 4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484) 
   * 27d751d0ef8d786c233bb4c16950a81f6b410e70 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-717095708


   Thanks for your pretty suggestive reviews @StephanEwen and I really get some profound, original insights from you!
   
   >  Compatibility with SSL
   
   I really overlooked the SSL branch before. After double verify, the file region way is indeed not compatible with SSL as you concerned, also mentioned in netty [issue](https://github.com/netty/netty/issues/2075). ATM I do not think of a proper way to support SSL with file region. Then my rough thought is to retain two branches to work around. The previous `FileBufferReader` implementation with memory overhead is still valid if SSL enabled. And I introduce another implementation of `BoundedData.Reader` for taking file region way without SSL option. 
   
   I guess SSL not widely used in practice, so most of the scenarios can still benefit from this improvement of memory overhead. WDYT or have other better options?
   
   > Naming
   
   I ever a bit torn whether to retain the previous naming for the reason you mentioned above. I totally agree with your saying `thinking of a file region as a lazy buffer`, then +1 to keep the previous naming.
   
   > An slightly different structure for Netty Messages
   
   After double checking, I had the misunderstanding of the netty stack call before to result in current implementation. I wrongly thought that `NettyMessage#write` can only write `FileRegion` instance without any `ByteBuf` instance in order to guarantee the followup `FileRegion#transferTo` called properly.  So I only allocated header buffer in `NettyMessage#write` and write it together with `FileRegion#transferTo`, that is the previous consideration without reusing `DefaultFileRegion` directly.
   
   Your above inputs of refactoring `NettyMessage#write` give me some insights to try out again and dispel my above misunderstanding. We can indeed write header ByteBuf to `ChannelOutboundInvoker` directly for both `BufferResponse` and `FileRegion` ways, then the code path is almost the same for them now. I conclude the following changes:
   
   - Introduce another `FileRegionResponse` which extends `DefaultFileRegion`, because we need to override the `#deallocate` method not to close file channel for every call.
   
   - Refactor current `NettyMessage#write` method into `write(ChannelOutboundInvoker out, ByteBufAllocator allocator, ChannelPromise promise)`, then return `void` for it. I find it will also bring some logics refactoring in some unit tests.
   
   - Make `BufferResponse#buffer` as an object which can be either `Buffer` or `FileRegionResponse`, but we also need to consider how to refactor current usages like `buffer#recycleBuffer`, `buffer#setAllocator` and `BufferResponse#getBuffer` after `buffer` becoming an object. 
   
   - Keep `NettyMessage` still as abstract class since we do not need to introduce extra `NettyMessage` subclass to reuse existing `BufferResponse` as above.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on a change in pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #13523:
URL: https://github.com/apache/flink/pull/13523#discussion_r516452223



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -371,6 +368,22 @@ ByteBuf write(ByteBufAllocator allocator) throws IOException {
 			}
 		}
 
+		private ByteBuf fillHeader(ByteBufAllocator allocator) {
+			// in order to forward the buffer to netty, it needs an allocator set
+			buffer.setAllocator(allocator);

Review comment:
       I ever had the same feeling before exactly as you said. The deduplication seems not worth here.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
##########
@@ -691,23 +672,12 @@ static CloseRequest readFrom(@SuppressWarnings("unused") ByteBuf buffer) throws
 		}
 
 		@Override
-		ByteBuf write(ByteBufAllocator allocator) throws IOException {
-			ByteBuf result = null;
-
-			try {
-				result = allocateBuffer(allocator, ID, 4 + 16);
-				result.writeInt(credit);
-				receiverId.writeTo(result);
-
-				return result;
-			}
-			catch (Throwable t) {
-				if (result != null) {
-					result.release();
-				}
-
-				throw new IOException(t);
-			}
+		void write(ChannelOutboundInvoker out, ChannelPromise promise, ByteBufAllocator allocator) throws IOException {
+			Consumer<ByteBuf> consumer = (bb) -> {

Review comment:
       I indeed overlooked the performance concern for the frequent message.  The `handleException` deduplication seems pretty good for me.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/FileRegionBuffer.java
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.buffer;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBuf;
+import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufAllocator;
+import org.apache.flink.shaded.netty4.io.netty.channel.DefaultFileRegion;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Wrapper for {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} instance.
+ *
+ * <p>The file region can be sent via network channel which supports zero-copy file transfer,
+ * and it can also be read into memory segment locally via {@link #readInto(MemorySegment)}.
+ */
+public class FileRegionBuffer extends DefaultFileRegion implements Buffer {
+
+	/** The number of bytes to be read/transferred from this file region. */
+	private final int bufferSize;
+
+	private final FileChannel fileChannel;
+
+	/** The {@link DataType} this buffer represents. */
+	private final DataType dataType;
+
+	/** Whether the buffer is compressed or not. */
+	private final boolean isCompressed;
+
+	public FileRegionBuffer(
+			FileChannel fileChannel,
+			int bufferSize,
+			DataType dataType,
+			boolean isCompressed) throws IOException {
+
+		super(fileChannel, fileChannel.position(), bufferSize);
+
+		this.fileChannel = checkNotNull(fileChannel);
+		this.bufferSize = bufferSize;
+		this.dataType = checkNotNull(dataType);
+		this.isCompressed = isCompressed;
+	}
+
+	// ------------------------------------------------------------------------
+	// Buffer override methods
+	// ------------------------------------------------------------------------
+
+	@Override
+	public boolean isBuffer() {
+		return dataType.isBuffer();
+	}
+
+	@Override
+	public MemorySegment getMemorySegment() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getMemorySegmentOffset() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public ReadOnlySlicedNetworkBuffer readOnlySlice() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public ReadOnlySlicedNetworkBuffer readOnlySlice(int index, int length) {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getMaxCapacity() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public int getReaderIndex() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void setReaderIndex(int readerIndex) throws IndexOutOfBoundsException {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	/**
+	 * This method is only implemented for tests at the moment.
+	 */
+	@Override
+	public ByteBuffer getNioBufferReadable() {
+		return BufferReaderWriterUtil.tryReadByteBuffer(fileChannel, bufferSize);

Review comment:
       As I mentioned in javadoc for this method, it is only used for unit tests ATM, not for the core codes.
   Actually in the first version I made this method for `UnsupportedOperationException`, but I found many exiting tests rely on this method like `BoundedBlockingSubpartitionWriteReadTest`. So I implemented this method for avoiding refactoring the logic of unit tests for less efforts. Do you think we should refactor the unit test to avoid calling this way instead?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferReaderWriterUtil.java
##########
@@ -187,6 +205,17 @@ static ByteBuffer allocatedHeaderBuffer() {
 		return new ByteBuffer[] { allocatedHeaderBuffer(), null };
 	}
 
+	public static ByteBuffer tryReadByteBuffer(FileChannel channel, int size) {
+		ByteBuffer bb = ByteBuffer.allocate(size);
+		try {
+			tryReadByteBuffer(channel, bb);
+			bb.flip();
+		} catch (IOException ex) {
+			ExceptionUtils.rethrow(ex);

Review comment:
       Agree, I really had the same concern before. Actually my previous way was throwing explicit `IOException` here. But I found it also need to adjust the interface method `Buffer#getNioBufferReadable` for throwing `IOException`, even it would further bring up many serious of changes for exiting usages.
   
   It was also caused by [comment](https://github.com/apache/flink/pull/13523#discussion_r516235008). Then let's confirm them together.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {
+
+	/** The result subpartition that we read. */
+	private final BoundedBlockingSubpartition parent;
+
+	/** The reader/decoder to the file region with the data we currently read from. */
+	private final BoundedData.Reader dataReader;
+
+	/** The remaining number of data buffers (not events) in the result. */
+	private int numDataBuffers;
+
+	/** The remaining number of data buffers and events in the result. */
+	private int numDataAndEventBuffers;
+
+	/** Flag whether this reader is released. */
+	private boolean isReleased;
+
+	private int sequenceNumber;
+
+	BoundedBlockingSubpartitionView(
+		BoundedBlockingSubpartition parent,
+		Path filePath,
+		int numDataBuffers,
+		int numDataAndEventBuffers) throws IOException {
+
+		this.parent = checkNotNull(parent);
+
+		checkNotNull(filePath);
+		this.dataReader = new FileRegionReader(filePath);
+
+		checkArgument(numDataBuffers >= 0);
+		this.numDataBuffers = numDataBuffers;
+
+		checkArgument(numDataAndEventBuffers >= 0);
+		this.numDataAndEventBuffers = numDataAndEventBuffers;
+	}
+
+	@Nullable
+	@Override
+	public BufferAndBacklog getNextBuffer() throws IOException {
+		if (isReleased) {
+			return null;
+		}
+
+		Buffer current = dataReader.nextBuffer();
+		if (current == null) {
+			// as per contract, we must return null when the reader is empty,
+			// but also in case the reader is disposed (rather than throwing an exception)
+			return null;
+		}
+
+		updateStatistics(current);
+
+		// We simply assume all the data are non-events for batch jobs to avoid pre-fetching the next header
+		Buffer.DataType nextDataType = numDataAndEventBuffers > 0 ? Buffer.DataType.DATA_BUFFER : Buffer.DataType.NONE;
+		return BufferAndBacklog.fromBufferAndLookahead(current, nextDataType, numDataBuffers, sequenceNumber++);
+	}
+
+	private void updateStatistics(Buffer buffer) {
+		if (buffer.isBuffer()) {
+			numDataBuffers--;
+		}
+		numDataAndEventBuffers--;
+	}
+
+	@Override
+	public boolean isAvailable(int numCreditsAvailable) {
+		// We simply assume there are no events except EndOfPartitionEvent for bath jobs,
+		// then it has no essential effect to ignore the judgement of next event buffer.
+		return numCreditsAvailable > 0 && numDataAndEventBuffers > 0;
+	}
+
+	@Override
+	public void releaseAllResources() throws IOException {
+		// it is not a problem if this method executes multiple times
+		isReleased = true;
+
+		IOUtils.closeQuietly(dataReader);
+
+		// Notify the parent that this one is released. This allows the parent to
+		// eventually release all resources (when all readers are done and the
+		// parent is disposed).
+		parent.releaseReaderReference(this);
+	}
+
+	@Override
+	public boolean isReleased() {
+		return isReleased;
+	}
+
+	@Override
+	public Throwable getFailureCause() {
+		// we can never throw an error after this was created
+		return null;
+	}
+
+	@Override
+	public int unsynchronizedGetNumberOfQueuedBuffers() {
+		return parent.unsynchronizedGetNumberOfQueuedBuffers();
+	}
+
+	@Override
+	public void notifyDataAvailable() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void resumeConsumption() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public String toString() {
+		return String.format("Blocking Subpartition Reader: ID=%s, index=%d",
+			parent.parent.getPartitionId(),
+			parent.getSubPartitionIndex());
+	}
+
+	/**
+	 * The reader to read from {@link BoundedBlockingSubpartition} and return the wrapped
+	 * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} based buffer.
+	 */
+	static final class FileRegionReader implements BoundedData.Reader {
+
+		private final FileChannel fileChannel;
+
+		private final ByteBuffer headerBuffer;
+
+		private long lastPosition = -1L;
+
+		private int lastBufferSize;
+
+		FileRegionReader(Path filePath) throws IOException {
+			this.fileChannel = FileChannel.open(filePath, StandardOpenOption.READ);
+			this.headerBuffer = BufferReaderWriterUtil.allocatedHeaderBuffer();
+		}
+
+		@Nullable
+		@Override
+		public Buffer nextBuffer() throws IOException {
+			// As the file region transfer via network channel will not modify the underlining file position,

Review comment:
       The current way is indeed a bit obscure. Actually it is involved in two different usages ATM:
   
   - For network file transfer, the underlining file position for data part is not changed while transferring.
   
   - For local input channel reading, it will change the respective underlining file position via `FileRegionBuffer#readInto`.
   
   So when the `nextBuffer` is called here, it does not know the caller is from local channel or network stack, then it can not always advance the position. Here I rely on the `lastPosition` for implicitly distinguishing the different usages.
   
   My first version PR seems a bit better here to always advance the position in the constructor of extended `FileRegion` instance, since it would be only used for network stack. But after the refactoring in the current version, I did not find another better place to advance the position reasonably. Or I misunderstood your proposed way?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BoundedBlockingSubpartitionView.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.partition.ResultSubpartition.BufferAndBacklog;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The reader (read view) of a BoundedBlockingSubpartition based on
+ * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion}.
+ */
+public class BoundedBlockingSubpartitionView implements ResultSubpartitionView {
+
+	/** The result subpartition that we read. */
+	private final BoundedBlockingSubpartition parent;
+
+	/** The reader/decoder to the file region with the data we currently read from. */
+	private final BoundedData.Reader dataReader;
+
+	/** The remaining number of data buffers (not events) in the result. */
+	private int numDataBuffers;
+
+	/** The remaining number of data buffers and events in the result. */
+	private int numDataAndEventBuffers;
+
+	/** Flag whether this reader is released. */
+	private boolean isReleased;
+
+	private int sequenceNumber;
+
+	BoundedBlockingSubpartitionView(
+		BoundedBlockingSubpartition parent,
+		Path filePath,
+		int numDataBuffers,
+		int numDataAndEventBuffers) throws IOException {
+
+		this.parent = checkNotNull(parent);
+
+		checkNotNull(filePath);
+		this.dataReader = new FileRegionReader(filePath);
+
+		checkArgument(numDataBuffers >= 0);
+		this.numDataBuffers = numDataBuffers;
+
+		checkArgument(numDataAndEventBuffers >= 0);
+		this.numDataAndEventBuffers = numDataAndEventBuffers;
+	}
+
+	@Nullable
+	@Override
+	public BufferAndBacklog getNextBuffer() throws IOException {
+		if (isReleased) {
+			return null;
+		}
+
+		Buffer current = dataReader.nextBuffer();
+		if (current == null) {
+			// as per contract, we must return null when the reader is empty,
+			// but also in case the reader is disposed (rather than throwing an exception)
+			return null;
+		}
+
+		updateStatistics(current);
+
+		// We simply assume all the data are non-events for batch jobs to avoid pre-fetching the next header
+		Buffer.DataType nextDataType = numDataAndEventBuffers > 0 ? Buffer.DataType.DATA_BUFFER : Buffer.DataType.NONE;
+		return BufferAndBacklog.fromBufferAndLookahead(current, nextDataType, numDataBuffers, sequenceNumber++);
+	}
+
+	private void updateStatistics(Buffer buffer) {
+		if (buffer.isBuffer()) {
+			numDataBuffers--;
+		}
+		numDataAndEventBuffers--;
+	}
+
+	@Override
+	public boolean isAvailable(int numCreditsAvailable) {
+		// We simply assume there are no events except EndOfPartitionEvent for bath jobs,
+		// then it has no essential effect to ignore the judgement of next event buffer.
+		return numCreditsAvailable > 0 && numDataAndEventBuffers > 0;
+	}
+
+	@Override
+	public void releaseAllResources() throws IOException {
+		// it is not a problem if this method executes multiple times
+		isReleased = true;
+
+		IOUtils.closeQuietly(dataReader);
+
+		// Notify the parent that this one is released. This allows the parent to
+		// eventually release all resources (when all readers are done and the
+		// parent is disposed).
+		parent.releaseReaderReference(this);
+	}
+
+	@Override
+	public boolean isReleased() {
+		return isReleased;
+	}
+
+	@Override
+	public Throwable getFailureCause() {
+		// we can never throw an error after this was created
+		return null;
+	}
+
+	@Override
+	public int unsynchronizedGetNumberOfQueuedBuffers() {
+		return parent.unsynchronizedGetNumberOfQueuedBuffers();
+	}
+
+	@Override
+	public void notifyDataAvailable() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public void resumeConsumption() {
+		throw new UnsupportedOperationException("Method should never be called.");
+	}
+
+	@Override
+	public String toString() {
+		return String.format("Blocking Subpartition Reader: ID=%s, index=%d",
+			parent.parent.getPartitionId(),
+			parent.getSubPartitionIndex());
+	}
+
+	/**
+	 * The reader to read from {@link BoundedBlockingSubpartition} and return the wrapped
+	 * {@link org.apache.flink.shaded.netty4.io.netty.channel.FileRegion} based buffer.
+	 */
+	static final class FileRegionReader implements BoundedData.Reader {
+
+		private final FileChannel fileChannel;
+
+		private final ByteBuffer headerBuffer;
+
+		private long lastPosition = -1L;
+
+		private int lastBufferSize;
+
+		FileRegionReader(Path filePath) throws IOException {
+			this.fileChannel = FileChannel.open(filePath, StandardOpenOption.READ);
+			this.headerBuffer = BufferReaderWriterUtil.allocatedHeaderBuffer();
+		}
+
+		@Nullable
+		@Override
+		public Buffer nextBuffer() throws IOException {
+			// As the file region transfer via network channel will not modify the underlining file position,

Review comment:
       Yeah, I agree with the fragile point. Let me consider how to refactor it a bit.
   
   The `FileRegionBuffer#readInto` relies on `FileChannel#read(ByteBuffer dst)` to read the data part, which will automatically updates the position. If I always advance the data part position here before actual reading, then for local channel reading, I need to roll back the respective position. It might be a bit better than current way since we do not need to maintain the `lastPosition` variable. My understanding is right?
   
   The realistic way is to only advance the position for network direct transfer, and leave the normal advance for local input channel while `FileChannel#read(ByteBuffer dst)`. I will further think whether it can be done in some place.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] zhijiangW commented on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-718688119


   Thanks for the further confirmation @StephanEwen !
   
   > Do we need to FileRegionResponse to extend DefaultFileRegion, or can the FileRegionResponse contain a DefaultFileRegion object? Given the complex interface of FileRegion in Netty (and how it may change between versions) might be good to avoid inheriting from it.
   
   As we confirmed before, we only need one `BufferResponse` message for both `Buffer` and `FileRegion` ways. I tried to introduce `FileRegionBuffer` to implement `Buffer` interface and it has the `FileRegion` inside. Then we do not need to touch any previous `Buffer` instances and compatible with relevant existing interfaces.
   
   But we can not reuse `DefaultFileRegion` provided by netty directly, since the `DefaultFileRegion#deallocate()` would close underlying `FileChannel` for every transfer. So we have to extend `DefaultFileRegion` to override `#deallocate` method with empty operation. 
   
   > You are right about buffer#recycleBuffer, buffer#setAllocator and BufferResponse#getBuffer - we need to handle this differently for buffers and regions.
   
   I guess we do not need consider these issues now, as I assume `FileRegionBuffer` implement `Buffer` interface compatible with existing methods. Also we do not need any `Object` field in `BufferResponse` component.
   
   > Handling SSL
   
   Totally agree with your points for it.
   
   FYI: My new version codes are almost ready to resolve all the above comments. I will update this PR probably early tomorrow after polishing the code a bit. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13523: [FLINK-15981][network] Implement FileRegion way to shuffle file-based blocking partition in network stack

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13523:
URL: https://github.com/apache/flink/pull/13523#issuecomment-701325670


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3461a87abdb961092e464030700a29964833de7a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7125",
       "triggerID" : "3461a87abdb961092e464030700a29964833de7a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7409",
       "triggerID" : "a4198d83d630c6efa49d0153142e201426bc6507",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7484",
       "triggerID" : "4a0bb7eabb3a209fbdd7b8d963d4da34b1b3f8ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7498",
       "triggerID" : "27d751d0ef8d786c233bb4c16950a81f6b410e70",
       "triggerType" : "PUSH"
     }, {
       "hash" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7520",
       "triggerID" : "84e51e22e60e09d1263dcc07495dc2dadd2f105e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605",
       "triggerID" : "975212a86b00dd07f3028e81760f8fe15b29b926",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 975212a86b00dd07f3028e81760f8fe15b29b926 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7605) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org