You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/19 09:51:27 UTC
[GitHub] [flink] pnowojski opened a new pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
pnowojski opened a new pull request #12243:
URL: https://github.com/apache/flink/pull/12243
It's possible that indexes of passed InputGates are not monotonic - that left input has higher input gate index. This commit fixes an ArrayIndexOutOfBound caused by this.
## Verifying this change
This PR adds a dedicated unit test.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r427415485
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/InputProcessorUtilTest.java
##########
@@ -58,4 +79,57 @@ public void testGenerateInputGateToChannelIndexOffsetMap() {
assertEquals(0, inputGateToChannelIndexOffsetMap.get(ig1).intValue());
assertEquals(3, inputGateToChannelIndexOffsetMap.get(ig2).intValue());
}
+
+ @Test
+ public void testCreateCheckpointedMultipleInputGate() throws Exception {
+ try (CloseableRegistry registry = new CloseableRegistry()) {
+ MockEnvironment environment = new MockEnvironmentBuilder().build();
+ MockStreamTask streamTask = new MockStreamTaskBuilder(environment).build();
+ StreamConfig streamConfig = new StreamConfig(environment.getJobConfiguration());
+ streamConfig.setCheckpointMode(CheckpointingMode.EXACTLY_ONCE);
+ streamConfig.setUnalignedCheckpointsEnabled(true);
+
+ // First input gate has index larger than the second
+ Collection<IndexedInputGate>[] inputGates = new Collection[] {
+ Collections.singletonList(new MockIndexedInputGate(1, 4)),
+ Collections.singletonList(new MockIndexedInputGate(0, 2)),
+ };
+
+ new MockChannelStateWriter() {
+ @Override
+ public void addInputData(
+ long checkpointId,
+ InputChannelInfo info,
+ int startSeqNum,
+ Buffer... data) {
+ super.addInputData(checkpointId, info, startSeqNum, data);
+ }
+ };
+
+ CheckpointedInputGate[] checkpointedMultipleInputGate = InputProcessorUtil.createCheckpointedMultipleInputGate(
+ streamTask,
+ streamConfig,
+ new MockChannelStateWriter(),
+ environment.getMetricGroup().getIOMetricGroup(),
+ streamTask.getName(),
+ inputGates);
+ for (CheckpointedInputGate checkpointedInputGate : checkpointedMultipleInputGate) {
+ registry.registerCloseable(checkpointedInputGate);
+ }
+
+ CheckpointBarrierHandler barrierHandler = checkpointedMultipleInputGate[0].getCheckpointBarrierHandler();
+
+ List<IndexedInputGate> allInputGates = Arrays.stream(inputGates).flatMap(gates -> gates.stream()).collect(Collectors.toList());
+ for (IndexedInputGate inputGate : allInputGates) {
+ for (int channelId = 0; channelId < inputGate.getNumberOfInputChannels(); channelId++) {
+ assertTrue(barrierHandler.getBufferReceivedListener().isPresent());
+ BufferReceivedListener bufferReceivedListener = barrierHandler.getBufferReceivedListener().get();
+ bufferReceivedListener.notifyBarrierReceived(
+ new CheckpointBarrier(1, 42, CheckpointOptions.forCheckpointWithDefaultLocation(true, true)),
+ new InputChannelInfo(inputGate.getGateIndex(), channelId));
+ }
+ }
+ assertFalse(barrierHandler.isCheckpointPending());
Review comment:
I am not sure why we need to verify this because it never process the barrier in this test. Maybe verify `assertTrue(barrierHandler.getAllBarriersReceivedFuture(1).isDone())` instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630715944
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit b956522108b0344ff004e859c0bc399dc8c38348 (Tue May 19 09:53:57 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
}, {
"hash" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b956522108b0344ff004e859c0bc399dc8c38348 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832)
* a3be362324a56a5f9b118a09ea3552a3039acffe UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b956522108b0344ff004e859c0bc399dc8c38348 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski merged pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
pnowojski merged pull request #12243:
URL: https://github.com/apache/flink/pull/12243
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r428017856
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/InputProcessorUtilTest.java
##########
@@ -58,4 +79,57 @@ public void testGenerateInputGateToChannelIndexOffsetMap() {
assertEquals(0, inputGateToChannelIndexOffsetMap.get(ig1).intValue());
assertEquals(3, inputGateToChannelIndexOffsetMap.get(ig2).intValue());
}
+
+ @Test
+ public void testCreateCheckpointedMultipleInputGate() throws Exception {
+ try (CloseableRegistry registry = new CloseableRegistry()) {
+ MockEnvironment environment = new MockEnvironmentBuilder().build();
+ MockStreamTask streamTask = new MockStreamTaskBuilder(environment).build();
+ StreamConfig streamConfig = new StreamConfig(environment.getJobConfiguration());
+ streamConfig.setCheckpointMode(CheckpointingMode.EXACTLY_ONCE);
+ streamConfig.setUnalignedCheckpointsEnabled(true);
+
+ // First input gate has index larger than the second
+ Collection<IndexedInputGate>[] inputGates = new Collection[] {
+ Collections.singletonList(new MockIndexedInputGate(1, 4)),
+ Collections.singletonList(new MockIndexedInputGate(0, 2)),
+ };
+
+ new MockChannelStateWriter() {
Review comment:
ops, that's a left over of some previous version.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/InputProcessorUtil.java
##########
@@ -79,11 +80,26 @@ public static CheckpointedInputGate createCheckpointedInputGate(
unionedInputGates[i] = InputGateUtil.createInputGate(inputGates[i].toArray(new IndexedInputGate[0]));
}
+ IntStream numberOfInputChannelsPerGate =
+ Arrays
+ .stream(inputGates)
+ .flatMap(collection -> collection.stream())
+ .sorted(Comparator.comparingInt(IndexedInputGate::getGateIndex))
+ .mapToInt(InputGate::getNumberOfInputChannels);
+
Map<InputGate, Integer> inputGateToChannelIndexOffset = generateInputGateToChannelIndexOffsetMap(unionedInputGates);
+ // Note that numberOfInputChannelsPerGate and inputGateToChannelIndexOffset have a bit different
Review comment:
Hmmm, I'm not sure, as what if left input has input gates with indexes `0` and `3`, while the right input has indexes `1`, `2` and `4`? (I'm not sure if that's a valid scenario in the JobGraphGenerator) Left input would have a one instance of `UnionInputGate` over gates 0 and 3, while right input would have another instance with gates 1, 2 and 4. However we sort them, it would be somehow inconsistent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
}, {
"hash" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950",
"triggerID" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3be362324a56a5f9b118a09ea3552a3039acffe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b956522108b0344ff004e859c0bc399dc8c38348 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
}, {
"hash" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950",
"triggerID" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"triggerType" : "PUSH"
}, {
"hash" : "67c3ff5187719aa012122785db68af1d9de77dc2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2014",
"triggerID" : "67c3ff5187719aa012122785db68af1d9de77dc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3be362324a56a5f9b118a09ea3552a3039acffe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950)
* 67c3ff5187719aa012122785db68af1d9de77dc2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2014)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630828142
Test failure is unrelated and fixed on more recent master by some commit (don't know which)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r427414666
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/InputProcessorUtilTest.java
##########
@@ -58,4 +79,57 @@ public void testGenerateInputGateToChannelIndexOffsetMap() {
assertEquals(0, inputGateToChannelIndexOffsetMap.get(ig1).intValue());
assertEquals(3, inputGateToChannelIndexOffsetMap.get(ig2).intValue());
}
+
+ @Test
+ public void testCreateCheckpointedMultipleInputGate() throws Exception {
+ try (CloseableRegistry registry = new CloseableRegistry()) {
+ MockEnvironment environment = new MockEnvironmentBuilder().build();
+ MockStreamTask streamTask = new MockStreamTaskBuilder(environment).build();
+ StreamConfig streamConfig = new StreamConfig(environment.getJobConfiguration());
+ streamConfig.setCheckpointMode(CheckpointingMode.EXACTLY_ONCE);
+ streamConfig.setUnalignedCheckpointsEnabled(true);
+
+ // First input gate has index larger than the second
+ Collection<IndexedInputGate>[] inputGates = new Collection[] {
+ Collections.singletonList(new MockIndexedInputGate(1, 4)),
+ Collections.singletonList(new MockIndexedInputGate(0, 2)),
+ };
+
+ new MockChannelStateWriter() {
+ @Override
+ public void addInputData(
+ long checkpointId,
+ InputChannelInfo info,
+ int startSeqNum,
+ Buffer... data) {
+ super.addInputData(checkpointId, info, startSeqNum, data);
+ }
+ };
+
+ CheckpointedInputGate[] checkpointedMultipleInputGate = InputProcessorUtil.createCheckpointedMultipleInputGate(
+ streamTask,
+ streamConfig,
+ new MockChannelStateWriter(),
+ environment.getMetricGroup().getIOMetricGroup(),
+ streamTask.getName(),
+ inputGates);
+ for (CheckpointedInputGate checkpointedInputGate : checkpointedMultipleInputGate) {
+ registry.registerCloseable(checkpointedInputGate);
+ }
+
+ CheckpointBarrierHandler barrierHandler = checkpointedMultipleInputGate[0].getCheckpointBarrierHandler();
+
+ List<IndexedInputGate> allInputGates = Arrays.stream(inputGates).flatMap(gates -> gates.stream()).collect(Collectors.toList());
+ for (IndexedInputGate inputGate : allInputGates) {
+ for (int channelId = 0; channelId < inputGate.getNumberOfInputChannels(); channelId++) {
+ assertTrue(barrierHandler.getBufferReceivedListener().isPresent());
Review comment:
nit: might be better to place these two lines out of loop to together with `barrierHandler`
```
CheckpointBarrierHandler barrierHandler = checkpointedMultipleInputGate[0].getCheckpointBarrierHandler();
assertTrue(barrierHandler.getBufferReceivedListener().isPresent());
BufferReceivedListener bufferReceivedListener = barrierHandler.getBufferReceivedListener().get();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Jiayi-Liao commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
Jiayi-Liao commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r428063945
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/InputProcessorUtil.java
##########
@@ -79,11 +80,26 @@ public static CheckpointedInputGate createCheckpointedInputGate(
unionedInputGates[i] = InputGateUtil.createInputGate(inputGates[i].toArray(new IndexedInputGate[0]));
}
+ IntStream numberOfInputChannelsPerGate =
+ Arrays
+ .stream(inputGates)
+ .flatMap(collection -> collection.stream())
+ .sorted(Comparator.comparingInt(IndexedInputGate::getGateIndex))
+ .mapToInt(InputGate::getNumberOfInputChannels);
+
Map<InputGate, Integer> inputGateToChannelIndexOffset = generateInputGateToChannelIndexOffsetMap(unionedInputGates);
+ // Note that numberOfInputChannelsPerGate and inputGateToChannelIndexOffset have a bit different
Review comment:
You're right. I didn't notice that `inputGateToChannelIndexOffset`'s key is an unioned InputGate. Thanks for pointing this out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1832",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
}, {
"hash" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950",
"triggerID" : "a3be362324a56a5f9b118a09ea3552a3039acffe",
"triggerType" : "PUSH"
}, {
"hash" : "67c3ff5187719aa012122785db68af1d9de77dc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "67c3ff5187719aa012122785db68af1d9de77dc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3be362324a56a5f9b118a09ea3552a3039acffe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1950)
* 67c3ff5187719aa012122785db68af1d9de77dc2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630723410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b956522108b0344ff004e859c0bc399dc8c38348",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b956522108b0344ff004e859c0bc399dc8c38348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b956522108b0344ff004e859c0bc399dc8c38348 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12243:
URL: https://github.com/apache/flink/pull/12243#issuecomment-630715944
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 67c3ff5187719aa012122785db68af1d9de77dc2 (Fri Oct 16 10:53:34 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Jiayi-Liao commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
Jiayi-Liao commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r427381013
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/InputProcessorUtil.java
##########
@@ -79,11 +80,26 @@ public static CheckpointedInputGate createCheckpointedInputGate(
unionedInputGates[i] = InputGateUtil.createInputGate(inputGates[i].toArray(new IndexedInputGate[0]));
}
+ IntStream numberOfInputChannelsPerGate =
+ Arrays
+ .stream(inputGates)
+ .flatMap(collection -> collection.stream())
+ .sorted(Comparator.comparingInt(IndexedInputGate::getGateIndex))
+ .mapToInt(InputGate::getNumberOfInputChannels);
+
Map<InputGate, Integer> inputGateToChannelIndexOffset = generateInputGateToChannelIndexOffsetMap(unionedInputGates);
+ // Note that numberOfInputChannelsPerGate and inputGateToChannelIndexOffset have a bit different
Review comment:
Yes... It took me a while to understand their usage. Why don't we make them consistent by easily sorting the `inputGates` before doing anything here (there's no risk if I understand correctly), so that it won't bother us in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #12243: [FLINK-17805][networ] Fix ArrayIndexOutOfBound for rotated input gate indexes
Posted by GitBox <gi...@apache.org>.
zhijiangW commented on a change in pull request #12243:
URL: https://github.com/apache/flink/pull/12243#discussion_r427413431
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/InputProcessorUtilTest.java
##########
@@ -58,4 +79,57 @@ public void testGenerateInputGateToChannelIndexOffsetMap() {
assertEquals(0, inputGateToChannelIndexOffsetMap.get(ig1).intValue());
assertEquals(3, inputGateToChannelIndexOffsetMap.get(ig2).intValue());
}
+
+ @Test
+ public void testCreateCheckpointedMultipleInputGate() throws Exception {
+ try (CloseableRegistry registry = new CloseableRegistry()) {
+ MockEnvironment environment = new MockEnvironmentBuilder().build();
+ MockStreamTask streamTask = new MockStreamTaskBuilder(environment).build();
+ StreamConfig streamConfig = new StreamConfig(environment.getJobConfiguration());
+ streamConfig.setCheckpointMode(CheckpointingMode.EXACTLY_ONCE);
+ streamConfig.setUnalignedCheckpointsEnabled(true);
+
+ // First input gate has index larger than the second
+ Collection<IndexedInputGate>[] inputGates = new Collection[] {
+ Collections.singletonList(new MockIndexedInputGate(1, 4)),
+ Collections.singletonList(new MockIndexedInputGate(0, 2)),
+ };
+
+ new MockChannelStateWriter() {
Review comment:
seems redundant codes `new MockChannelStateWriter()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org