You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/11 03:20:29 UTC
[GitHub] [flink] curcur opened a new pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832
## What is the purpose of the change
`Execution#producedPartitions` are registered through an asynchronous interface `ShuffleMaster#registerPartitionWithProducer` to `ShuffleMaster`, however they are not always accessed through callbacks. So, it is possible that `Execution#producedPartitions` have not been available yet when accessed (in `Execution#deploy` for example). Since the only implementation of `ShuffleMaster` is `NettyShuffleMaster`, which indeed registers producedPartition in a synchronous way, hence this pull request enforces synchronous registrations under an asynchronous interface as a temporary fix to prevent future misuses.
## Brief change log
Make sure the future of registration is done immediately after registration. Throws IllegalStateException otherwise.
## Verifying this change
unit test:
ExecutionTest#testSynchronousRegisterProducedPartitions
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? JavaDocs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366683268
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -398,6 +401,10 @@ public static JobVertex createJobVertex(String task1, int numTasks, Class<NoOpIn
return groupVertex;
}
+ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Exception {
Review comment:
Yeah, I think that a good idea to make the naming consistent.
However, there might be too many places to change, and might be good to open a separate (hot fix) PR for this purpose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367468699
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
Review comment:
I prefer `try/catch` blocks since they guarantee the exception occurs where we want it to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367223282
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
synced up offline and resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367466238
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
Review comment:
```suggestion
* Tests that we reject incompleted futures returned by {@link ShuffleMaster#registerPartitionWithProducer...}.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574470098
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367889955
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
Review comment:
```
try {
Execution.registerProducedPartitions(sourceVertex, new LocalTaskManagerLocation(), new ExecutionAttemptID(), false);
Assert.fail("expected exception");
} catch (IllegalStateException e) {
// expected
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol merged pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol merged pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367783969
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
Review comment:
by `try/catch`
I guess you mean something like this?
```
boolean incompletePartitionRegistrationFuture = false;
try {
Execution.registerProducedPartitions(sourceVertex, new LocalTaskManagerLocation(), new ExecutionAttemptID(), false);
} catch (IllegalStateException e) {
incompletePartitionRegistrationFuture = true;
}
assertTrue(incompletePartitionRegistrationFuture);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366214994
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -431,6 +433,18 @@ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism) {
Review comment:
Removed Mockito.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367472847
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+ final JobGraph jobGraph = new JobGraph("job graph");
+ final JobVertex source = new JobVertex("source");
+ final JobVertex target = new JobVertex("target");
+
+ source.setInvokableClass(AbstractInvokable.class);
+ target.setInvokableClass(AbstractInvokable.class);
+ target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
+ jobGraph.addVertex(source);
+ jobGraph.addVertex(target);
+ ExecutionGraph executionGraph = TestingExecutionGraphBuilder
+ .newBuilder()
+ .setJobGraph(jobGraph)
+ .setShuffleMaster(shuffleMaster)
+ .build();
+
+ final ExecutionVertex sourceVertex = executionGraph.getAllVertices().get(source.getID()).getTaskVertices()[0];
+ final TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(), InetAddress.getLoopbackAddress(), 1);
+ final ExecutionAttemptID attemptID = new ExecutionAttemptID();
+ Execution.registerProducedPartitions(sourceVertex, location, attemptID, EAGER.allowLazyDeployment());
Review comment:
Pretty roundabout way of defining the deployment mode; is it even relevant for this test?
```suggestion
Execution.registerProducedPartitions(sourceVertex, new LocalTaskManagerLocation(), new ExecutionAttemptID(), false);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365749861
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -431,6 +433,18 @@ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism) {
Review comment:
This is reused from ExecutionVertexDeploymentTest.
Let me have a try to see whether I can avoid Mockito and at the same time not to make the code too complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365723610
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
});
}
+ /**
+ * Register producedPartitions to {@link ShuffleMaster}
+ *
+ * <p>HACK: Please notice that this method simulates asynchronous registration in a synchronous way
+ * by making sure the returned {@link CompletableFuture} from {@link ShuffleMaster#registerPartitionWithProducer}
+ * is done immediately.
+ *
+ * <p>{@link Execution#producedPartitions} are registered through an asynchronous interface
+ * {@link ShuffleMaster#registerPartitionWithProducer} to {@link ShuffleMaster}, however they are not always
+ * accessed through callbacks. So, it is possible that {@link Execution#producedPartitions}
+ * have not been available yet when accessed (in {@link Execution#deploy} for example).
+ *
+ * <p>Since the only implementation of {@link ShuffleMaster} is {@link NettyShuffleMaster},
+ * which indeed registers producedPartition in a synchronous way, hence this method enforces
+ * synchronous registration under an asynchronous interface for now.
+ *
+ * <p>If asynchronous registration is needed in the future, use callbacks to access {@link Execution#producedPartitions}.
Review comment:
This is a temporary fix. So we'd better have a TODO here as a reminder for follow up fixes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366682089
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ ScheduledExecutorService executor = new DirectScheduledExecutorService();
Review comment:
This is not a must, but I'd still suggest to try to mark the variables final if possible, like we do in other parts in this test class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367744319
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
Review comment:
How about this:
Tests that asynchronous registrations (incomplete futures returned by {@link ShuffleMaster#registerPartitionWithProducer}) are rejected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365749861
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -431,6 +433,18 @@ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism) {
Review comment:
This is reused from ExecutionVertexDeploymentTest.
Let me have a try to see whether we can avoid Mockito and at the same time not to make the code too complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-575845406
Thanks for reviewing and merging! @zentol
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366830175
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I am kind of thinking it is an overkill to use the TestingExecutionGraphBuilder, which translate a JobGraph -> ExecutionGraph.
In this particular case, I need to test two ExecutionVertex with one having partition registered and verify if the registration is done in an asynchronous way, throw an exception otherwise. It is not testing the entire workflow, so whether to construct a complete graph does not matter that much.
More specifically, I need to retain the topological relationship between source and target for ExecutionVertex. But in an ExecutionGraph, I only have access to ExecutionVertex through `Iterable<ExecutionVertex> getAllExecutionVertices()`. Then I have to reconstruct the topological relation.
Down to code level, by using `TestingExecutionGraphBuilder`, which builds ExecutionGraph from JobGraph similar like this:
JobGraph jobGraph = new JobGraph("The graph");
JobVertex source = new JobVertex("source");
JobVertex target = new JobVertex("target");
target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
jobGraph.addVertex(source);
jobGraph.addVertex(target);
ExecutionGraph eg = TestingExecutionGraphBuilder
.newBuilder()
.setJobGraph(jobGraph)
.setShuffleMaster(shuffleMaster).build();
Then I have to identify which one is the source and which one is the target, and register with responding Execution.
The previous implementation looks more clean to me. What do you think? Or you have better way to do this.
> I think we should create an ExecutionGraph and get the vertices from it rather than using this `getExecutionVertex(...)` to create vertices. Otherwise these 2 vertices are actually from 2 different ExecutionGraphs.
> In this way you do not need to create edges manually. You can leverage TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365721185
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##########
@@ -31,6 +31,10 @@
/**
* Asynchronously register a partition and its producer with the shuffle service.
*
+ * <p>IMPORTANT: Please implement the asynchronous interface in a synchronous way.
+ * The default scheduler only supports synchronous registration for now.
+ * Please refer to {@link NettyShuffleMaster#registerPartitionWithProducer} for example.
Review comment:
I'd prefer to state that "one must return a completed future at the moment". It is clear enough so the user do need to refer to NettyShuffleMaster for the implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366734687
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I think we should create an ExecutionGraph and get the vertices from it rather than using this `getExecutionVertex(...)` to create vertices. Otherwise these 2 vertices are actually from 2 different ExecutionGraphs.
In this way you do not need to create edges manually. You can leverage TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366830175
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I am kind of thinking it is an overkill to use the TestingExecutionGraphBuilder, which translate a JobGraph -> ExecutionGraph.
In this particular case, I need to test two ExecutionVertex with one having partition registered and verify if the registration is done in an asynchronous way, throw an exception otherwise. It is not testing the entire workflow, so whether to construct a complete graph does not matter that much.
More specifically, I need to retain the topological relationship between source and target for ExecutionVertex. But in an ExecutionGraph, I only have access to ExecutionVertex through `Iterable<ExecutionVertex> getAllExecutionVertices()`. Then I have to reconstruct the topological relation.
Down to code level, by using `TestingExecutionGraphBuilder`, which builds ExecutionGraph from JobGraph similar like this:
```
JobGraph jobGraph = new JobGraph("The graph");
JobVertex source = new JobVertex("source");
JobVertex target = new JobVertex("target");
target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
jobGraph.addVertex(source);
jobGraph.addVertex(target);
ExecutionGraph eg = TestingExecutionGraphBuilder
.newBuilder()
.setJobGraph(jobGraph)
.setShuffleMaster(shuffleMaster)
.build();
```
Then I have to identify which one is the source and which one is the target, and register with responding Execution.
The previous implementation looks more clean to me. What do you think? Or you have better way to do this.
> I think we should create an ExecutionGraph and get the vertices from it rather than using this `getExecutionVertex(...)` to create vertices. Otherwise these 2 vertices are actually from 2 different ExecutionGraphs.
> In this way you do not need to create edges manually. You can leverage TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367744181
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
Review comment:
How about this:
Tests that asynchronous registrations (incomplete futures returned by {@link ShuffleMaster#registerPartitionWithProducer}) are rejected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366681570
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -416,20 +431,17 @@ public static ExecutionJobVertex getExecutionVertex(
jobGraph.setScheduleMode(scheduleMode);
ExecutionGraph graph = TestingExecutionGraphBuilder
- .newBuilder()
- .setJobGraph(jobGraph)
- .setIoExecutor(executor)
- .setFutureExecutor(executor)
- .build();
+ .newBuilder()
Review comment:
One indentation from the initial line is expected here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur closed pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur closed pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367744319
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
Review comment:
How about this:
Tests that asynchronous registrations (incomplete futures returned by {@link ShuffleMaster#registerPartitionWithProducer}) are rejected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-575588477
When you make changes to a PR please push them as a separate commit; makes it easier to figure out what has changed ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574088324
1. get rid of Mockito
2. address other comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574088324
Thanks @zhuzhurk for reviewing the PR.
Updated the PR
1. get rid of Mockito
2. address other comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574469507
@flinkbot run azure re-run
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365745754
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
+ TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(), InetAddress.getLoopbackAddress(), 1);
+ ExecutionAttemptID attemptID = new ExecutionAttemptID();
+ Execution.registerProducedPartitions(vertex, location, attemptID, EAGER.allowLazyDeployment());
+ }
+
+ @Nonnull
+ private ExecutionJobVertex getExecutionVertex(
Review comment:
that's a good idea, will do it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144889655 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144889655) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420)
* 65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367473419
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
});
}
+ /**
+ * Register producedPartitions to {@link ShuffleMaster}
+ *
+ * <p>HACK: Please notice that this method simulates asynchronous registration in a synchronous way
+ * by making sure the returned {@link CompletableFuture} from {@link ShuffleMaster#registerPartitionWithProducer}
+ * is done immediately.
+ *
+ * <p>{@link Execution#producedPartitions} are registered through an asynchronous interface
+ * {@link ShuffleMaster#registerPartitionWithProducer} to {@link ShuffleMaster}, however they are not always
+ * accessed through callbacks. So, it is possible that {@link Execution#producedPartitions}
+ * have not been available yet when accessed (in {@link Execution#deploy} for example).
+ *
+ * <p>Since the only implementation of {@link ShuffleMaster} is {@link NettyShuffleMaster},
+ * which indeed registers producedPartition in a synchronous way, hence this method enforces
Review comment:
```suggestion
* which indeed registers producedPartition in a synchronous way, this method enforces
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367783969
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
Review comment:
by `try/catch`
I guess you mean something like this?
```
boolean incompletePartitionRegistrationFuture = false;
try {
Execution.registerProducedPartitions(sourceVertex, new LocalTaskManagerLocation(), new ExecutionAttemptID(), false);
} catch (IllegalStateException e) {
incompletePartitionRegistrationFuture = true;
}
assertTrue(incompletePartitionRegistrationFuture);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r368188645
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
Review comment:
I see, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365729537
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
+ TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(), InetAddress.getLoopbackAddress(), 1);
+ ExecutionAttemptID attemptID = new ExecutionAttemptID();
+ Execution.registerProducedPartitions(vertex, location, attemptID, EAGER.allowLazyDeployment());
+ }
+
+ @Nonnull
+ private ExecutionJobVertex getExecutionVertex(
Review comment:
Could we extend ExecutionGraphTestUtils#getExecutionVertex(...) to customize shuffleMaster and reuse it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur removed a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur removed a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573614132
> Thanks for opening the PR @curcur . The change generally looks good to me.
> I have a few comments for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367475499
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
});
}
+ /**
+ * Register producedPartitions to {@link ShuffleMaster}
+ *
+ * <p>HACK: Please notice that this method simulates asynchronous registration in a synchronous way
+ * by making sure the returned {@link CompletableFuture} from {@link ShuffleMaster#registerPartitionWithProducer}
+ * is done immediately.
+ *
+ * <p>{@link Execution#producedPartitions} are registered through an asynchronous interface
+ * {@link ShuffleMaster#registerPartitionWithProducer} to {@link ShuffleMaster}, however they are not always
+ * accessed through callbacks. So, it is possible that {@link Execution#producedPartitions}
+ * have not been available yet when accessed (in {@link Execution#deploy} for example).
+ *
+ * <p>Since the only implementation of {@link ShuffleMaster} is {@link NettyShuffleMaster},
+ * which indeed registers producedPartition in a synchronous way, hence this method enforces
+ * synchronous registration under an asynchronous interface for now.
+ *
+ * <p>TODO: If asynchronous registration is needed in the future, use callbacks to access {@link Execution#producedPartitions}.
+ *
+ * @return a CompletableFuture for registered producedPartitions. In the current version, the CompletableFuture has already completed.
Review comment:
```suggestion
* @return a completed future of partition deployment descriptors
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573275694
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit f327a0f2cf227e32c6c46fd0a698117d972a35eb (Sat Jan 11 03:22:23 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367476432
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -626,6 +647,8 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
.getShuffleMaster()
.registerPartitionWithProducer(partitionDescriptor, producerDescriptor);
Review comment:
```suggestion
// temporary hack; the scheduler does not handle incomplete futures properly
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur removed a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur removed a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574469507
@flinkbot run azure re-run
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366685145
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -398,6 +401,10 @@ public static JobVertex createJobVertex(String task1, int numTasks, Class<NoOpIn
return groupVertex;
}
+ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Exception {
Review comment:
I did a try and seems the change is not that large.
I'm fine you keep it as is for now since it is out of the scope of this PR. I can apply a patch to rename it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367467557
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
Review comment:
```suggestion
public void testIncompletePartitionRegistrationFutureIsRejected() throws Exception {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366683268
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -398,6 +401,10 @@ public static JobVertex createJobVertex(String task1, int numTasks, Class<NoOpIn
return groupVertex;
}
+ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Exception {
Review comment:
Yeah, I think that a good idea to make the naming consistent.
However, there might be too many places to change, and might be good to open a separate PR for this purpose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365745182
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
Review comment:
Yep, this part is needed, there has to be one consumer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366830175
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I am kind of thinking it is an overkill to use the TestingExecutionGraphBuilder, which translate a JobGraph -> ExecutionGraph.
In this particular case, I need to test two ExecutionVertex with one having partition registered and verify if the registration is done in an asynchronous way, throw an exception otherwise. It is not testing the entire workflow, so whether to construct a complete graph does not matter that much.
More specifically, I need to retain the topological relationship between source and target for ExecutionVertex. But in an ExecutionGraph, I only have access to ExecutionVertex through `Iterable<ExecutionVertex> getAllExecutionVertices()`. Then I have to reconstruct the topological relation.
Down to code level, by using `TestingExecutionGraphBuilder`, which builds ExecutionGraph from JobGraph similar like this:
```
JobGraph jobGraph = new JobGraph("The graph");
JobVertex source = new JobVertex("source");
JobVertex target = new JobVertex("target");
target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
jobGraph.addVertex(source);
jobGraph.addVertex(target);
ExecutionGraph eg = TestingExecutionGraphBuilder
.newBuilder()
.setJobGraph(jobGraph)
.setShuffleMaster(shuffleMaster)
.build();
```
Then I have to identify which one is the source and which one is the target, and register with responding Execution.
The previous implementation looks more clean to me. What do you think? Or you have a better way to do this.
> I think we should create an ExecutionGraph and get the vertices from it rather than using this `getExecutionVertex(...)` to create vertices. Otherwise these 2 vertices are actually from 2 different ExecutionGraphs.
> In this way you do not need to create edges manually. You can leverage TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144889655 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144895043 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144889655) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420)
* 65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144895043) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367759810
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
});
}
+ /**
+ * Register producedPartitions to {@link ShuffleMaster}
+ *
+ * <p>HACK: Please notice that this method simulates asynchronous registration in a synchronous way
+ * by making sure the returned {@link CompletableFuture} from {@link ShuffleMaster#registerPartitionWithProducer}
+ * is done immediately.
+ *
+ * <p>{@link Execution#producedPartitions} are registered through an asynchronous interface
+ * {@link ShuffleMaster#registerPartitionWithProducer} to {@link ShuffleMaster}, however they are not always
+ * accessed through callbacks. So, it is possible that {@link Execution#producedPartitions}
+ * have not been available yet when accessed (in {@link Execution#deploy} for example).
+ *
+ * <p>Since the only implementation of {@link ShuffleMaster} is {@link NettyShuffleMaster},
+ * which indeed registers producedPartition in a synchronous way, hence this method enforces
Review comment:
good catch :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144889655 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144889655) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-575012464
Thank you so much @zhuzhurk for reviewing the code and providing great suggestions and feedback!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367891697
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##########
@@ -31,6 +31,10 @@
/**
* Asynchronously register a partition and its producer with the shuffle service.
*
+ * <p>IMPORTANT: the returned future must be completed due to limitations in the default scheduler.
+ * The default scheduler only supports synchronous registration at the moment.
Review comment:
This line and the one below effectively do not provide any new information; I'd remove them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144889655 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144895043 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144889655) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420)
* 65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144895043) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367473190
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
});
}
+ /**
+ * Register producedPartitions to {@link ShuffleMaster}
+ *
+ * <p>HACK: Please notice that this method simulates asynchronous registration in a synchronous way
+ * by making sure the returned {@link CompletableFuture} from {@link ShuffleMaster#registerPartitionWithProducer}
+ * is done immediately.
Review comment:
```suggestion
* is completed immediately.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367474772
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -626,6 +647,8 @@ public void setInitialState(@Nullable JobManagerTaskRestore taskRestore) {
.getShuffleMaster()
.registerPartitionWithProducer(partitionDescriptor, producerDescriptor);
+ Preconditions.checkState(shuffleDescriptorFuture.isDone(), "ProducedPartitions are expected to be registered and accessible");
Review comment:
```suggestion
Preconditions.checkState(shuffleDescriptorFuture.isDone(), "ShuffleDescriptor future was incomplete.");
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366681405
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -398,6 +401,10 @@ public static JobVertex createJobVertex(String task1, int numTasks, Class<NoOpIn
return groupVertex;
}
+ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Exception {
Review comment:
I know that ExecutionGraphTestUtils did have several `getExecutionVertex (...)` methods to map `JobVertexID` to `ExecutionJobVertex `, but I think it's better we rename the old and newly added ones to `getExecutionJobVertex (...)`. Because `ExecutionVertex` is a structure different from `ExecutionJobVertex`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144675097 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388 TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:66afb2a54d5da715b3d59bdf58a4de12af833c2f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144866402 TriggerType:PUSH TriggerID:66afb2a54d5da715b3d59bdf58a4de12af833c2f
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:f2d45fd25663092d9a11e0798900fb886ed46ba3 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144889655 TriggerType:PUSH TriggerID:f2d45fd25663092d9a11e0798900fb886ed46ba3
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
Hash:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144895043 TriggerType:PUSH TriggerID:65dacfec0ce1dd50c778865b9a91c7b2d2268eb6
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144675097) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4388)
* 66afb2a54d5da715b3d59bdf58a4de12af833c2f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144866402) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4416)
* f2d45fd25663092d9a11e0798900fb886ed46ba3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144889655) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4420)
* 65dacfec0ce1dd50c778865b9a91c7b2d2268eb6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144895043) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4423)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367744181
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
Review comment:
How about this:
Tests that asynchronous registrations (incomplete futures returned by {@link ShuffleMaster#registerPartitionWithProducer}) are rejected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:MANUAL TriggerID:574470098
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:MANUAL TriggerID:574470098
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144458002 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:fe731e4a039fba606be83a9535841c157a048d58 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355 TriggerType:PUSH TriggerID:fe731e4a039fba606be83a9535841c157a048d58
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144670693 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:1278b4488cb8bc05addaf359458ed4bae6c1967b Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386 TriggerType:PUSH TriggerID:1278b4488cb8bc05addaf359458ed4bae6c1967b
Hash:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:3eba7e89df2c0b5ad6ba098aaabbab42be37df2f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
* fe731e4a039fba606be83a9535841c157a048d58 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144458002) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4355)
* 1278b4488cb8bc05addaf359458ed4bae6c1967b Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144670693) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4386)
* 3eba7e89df2c0b5ad6ba098aaabbab42be37df2f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365720350
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##########
@@ -431,6 +433,18 @@ public static ExecutionJobVertex getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism) {
Review comment:
We are trying to get rid of Mockito in flink tests.
If we are touching this part, I'd prefer to introduce a mock ExecutionEdge class for this purpose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573614132
> Thanks for opening the PR @curcur . The change generally looks good to me.
> I have a few comments for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-574088324
Thanks @zhuzhurk for reviewing the PR.
Updated the PR with
1. get rid of Mockito
2. address other comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-575012464
Thank you so much @zhuzhurk for reviewing the code and providing great suggestions and feedbacks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-575495502
Thanks for reviewing the code @zentol ! Here is the updated version. Please let me know what do you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366830175
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
I am kind of thinking it is an overkill to use the TestingExecutionGraphBuilder, which translate a JobGraph -> ExecutionGraph.
In this particular case, I need to test two ExecutionVertex with one having partition registered and verify if the registration is done in an asynchronous way, throw an exception otherwise. It is not testing the entire workflow, so whether to construct a complete graph does not matter that much.
More specifically, I need to retain the topological relationship between source and target for ExecutionVertex. But in an ExecutionGraph, I only have access to ExecutionVertex through `Iterable<ExecutionVertex> getAllExecutionVertices()`. Then I have to reconstruct the topological relation.
Down to code level, by using `TestingExecutionGraphBuilder`, which builds ExecutionGraph from JobGraph similar like this:
```
JobGraph jobGraph = new JobGraph("The graph");
JobVertex source = new JobVertex("source");
JobVertex target = new JobVertex("target");
target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
jobGraph.addVertex(source);
jobGraph.addVertex(target);
ExecutionGraph eg = TestingExecutionGraphBuilder
.newBuilder()
.setJobGraph(jobGraph)
.setShuffleMaster(shuffleMaster).build();
```
Then I have to identify which one is the source and which one is the target, and register with responding Execution.
The previous implementation looks more clean to me. What do you think? Or you have better way to do this.
> I think we should create an ExecutionGraph and get the vertices from it rather than using this `getExecutionVertex(...)` to create vertices. Otherwise these 2 vertices are actually from 2 different ExecutionGraphs.
> In this way you do not need to create edges manually. You can leverage TestingExecutionGraphBuilder to build an ExecutionGraph and set the wanted shuffle master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r366844097
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +553,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+
+ final ExecutionJobVertex sourceJobVertex = getExecutionVertex(new JobVertexID(), executor, EAGER, shuffleMaster);
Review comment:
Currently we we have no other way to create an ExecutionVertex without creating an `ExecutionGraph`. So even if you use `ExecutionGraphTestUtils#getExecutionVertex(...)`, it still creates an `ExecutionGraph` for each vertex to achieve the goal, which is tricky that we are operating vertices on 2 different graphs and connect them with edges.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur opened a new pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832
## What is the purpose of the change
`Execution#producedPartitions` are registered through an asynchronous interface `ShuffleMaster#registerPartitionWithProducer` to `ShuffleMaster`, however they are not always accessed through callbacks. So, it is possible that `Execution#producedPartitions` have not been available yet when accessed (in `Execution#deploy` for example). Since the only implementation of `ShuffleMaster` is `NettyShuffleMaster`, which indeed registers producedPartition in a synchronous way, this pull request enforces synchronous registrations under an asynchronous interface as a temporary fix to prevent future misuses.
## Brief change log
- Make sure the future of registration is done immediately after registration.
- Throws IllegalStateException otherwise.
## Verifying this change
unit test:
ExecutionTest#testSynchronousRegisterProducedPartitions
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? JavaDocs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#issuecomment-573276729
<!--
Meta data
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:5eb1599945f1bee35c342f762cc25684013d2d83 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143984465 TriggerType:PUSH TriggerID:5eb1599945f1bee35c342f762cc25684013d2d83
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:73579bbc6b556fe42e55e21581c5994f84480843 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144033919 TriggerType:PUSH TriggerID:73579bbc6b556fe42e55e21581c5994f84480843
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/144299177 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:e303d84258f1e63edd45867f6c4e89e181307ee5 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325 TriggerType:PUSH TriggerID:e303d84258f1e63edd45867f6c4e89e181307ee5
Hash:426112c59a86ec127040178efda1085231f1988f Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
Hash:426112c59a86ec127040178efda1085231f1988f Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144326556 TriggerType:PUSH TriggerID:426112c59a86ec127040178efda1085231f1988f
-->
## CI report:
* 5eb1599945f1bee35c342f762cc25684013d2d83 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143984465) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4264)
* 73579bbc6b556fe42e55e21581c5994f84480843 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144033919) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4267)
* e303d84258f1e63edd45867f6c4e89e181307ee5 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144299177) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4325)
* 426112c59a86ec127040178efda1085231f1988f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144326556) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4334)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367471726
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +551,46 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ final ShuffleMaster<ShuffleDescriptor> shuffleMaster = new TestingShuffleMaster();
+ final JobGraph jobGraph = new JobGraph("job graph");
+ final JobVertex source = new JobVertex("source");
+ final JobVertex target = new JobVertex("target");
+
+ source.setInvokableClass(AbstractInvokable.class);
+ target.setInvokableClass(AbstractInvokable.class);
+ target.connectNewDataSetAsInput(source, POINTWISE, PIPELINED);
+ jobGraph.addVertex(source);
+ jobGraph.addVertex(target);
+ ExecutionGraph executionGraph = TestingExecutionGraphBuilder
+ .newBuilder()
+ .setJobGraph(jobGraph)
+ .setShuffleMaster(shuffleMaster)
+ .build();
+
+ final ExecutionVertex sourceVertex = executionGraph.getAllVertices().get(source.getID()).getTaskVertices()[0];
+ final TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(), InetAddress.getLoopbackAddress(), 1);
Review comment:
use `LocalTaskManagerLocation` instead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r367474323
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##########
@@ -31,6 +31,10 @@
/**
* Asynchronously register a partition and its producer with the shuffle service.
*
+ * <p>IMPORTANT: Please implement the asynchronous interface in a synchronous way.
Review comment:
Replace block with:
```suggestion
* <p>IMPORTANT: The returned future must be completed due to limitations in the current scheduler.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365732020
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##########
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
});
}
+ /**
+ * Tests that producedPartitions are registered synchronously under an asynchronous interface.
+ */
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
Review comment:
Are these lines necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services