You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/05 10:01:06 UTC
[GitHub] [flink] echauchot opened a new pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
echauchot opened a new pull request #14558:
URL: https://github.com/apache/flink/pull/14558
## What is the purpose of the change
Make _MiniClusterJobClient#getAccumulators()_ non-blocking in Streaming mode
## Brief change log
Get the serialized accumulators from the execution graph rather than getting the accumulators from the _JobExecutionResult_.
## Verifying this change
Changed _AccumulatorLiveITCase_ so that the verification of the accumulators is done by calling _getAccumulators()_ with both _ClusterClient_ (as it was in previous test version) and _MiniClusterJobClient_
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? it was a bug that was not compliant with the javadocs. Now it is, so javadoc is untouched by this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760141274
@rmetzger this PR is ready for review PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-756811406
@rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ savepoint tests, I know we need to get the accumulators from _JobExecutionResult_ for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know Flink enough yet, can you give me a clue on this condition?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5543d0dc0cc1d54619579ea2fdb855aac3085978 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #14558:
URL: https://github.com/apache/flink/pull/14558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot removed a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot removed a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760130203
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754536948
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf (Fri May 28 08:14:28 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14558:
URL: https://github.com/apache/flink/pull/14558#discussion_r555597955
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniClusterJobClient.java
##########
@@ -106,7 +108,25 @@ public JobID getJobID() {
@Override
public CompletableFuture<Map<String, Object>> getAccumulators() {
- return getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults);
+ //TODO: this is not the only case when we need to get the accumulators from JobExecutionResult.
+ // It is needed also for UnalignedCheckpointCompatibilityITCase savepoints tests to pass.
+ // What is the complete condition ?
+ if (!miniCluster.isRunning()) {
Review comment:
I'm wondering if the `miniCluster.isRunning()` might not be the proper condition ? Since a mini-cluster could be viewed as a cluster that could run multiple jobs, the miniCluster would still be able to run after the job is done logically. Perhaps we could use `jobResultFuture.isDone()` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754536948
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 44801100ae245f2cd0d10d24830552b87164f072 (Tue Jan 05 10:03:45 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44801100ae245f2cd0d10d24830552b87164f072 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645)
* 5543d0dc0cc1d54619579ea2fdb855aac3085978 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-756811406
@rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ I know we need to get the accumulators from _JobExecutionResult_ it that case for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know flink enough yet, can you give me a clue on this condition?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* c875ca96e006d84d17bf744cf2894954870ec194 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
}, {
"hash" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058",
"triggerID" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-768230246
@echauchot I'm really sorry for the delay. I'll take a look at this PR today!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
}, {
"hash" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058",
"triggerID" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* c875ca96e006d84d17bf744cf2894954870ec194 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051)
* aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
}, {
"hash" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058",
"triggerID" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c875ca96e006d84d17bf744cf2894954870ec194 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051)
* aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12058)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-765397114
Friendly ping: @rmetzger do you have time to review/merge this PR? if not, can you ping someone to look ? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760131077
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760135401
@flinkbot run travis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760129111
> > @rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ savepoint tests, I know we need to get the accumulators from _JobExecutionResult_ for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know Flink enough yet, can you give me a clue on this condition?
>
> Hi @echauchot, from my point of view, there would be two possible semantics when users want to acquire accumulators:
>
> 1. Users want to retrieve the current value of accumulators immediately.
> 2. Users knows the job is bounded and want to acquire the final accumulator values after the job is finished.
>
> Previously `getAccumulator()` takes the second semantics, and now we want to move it to the first semantics to make it consistent with the other client implementation. Therefore, if users still want to use the second semantics, I think they would need to call `getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)` explicitly. Thus for me to fix the failed tests we need to modify the tests since we would not be able to distinguish the first and second semantics automatically. What do you think about it ?
@gaoyunhaii thanks for your comments. I totally agree with them. I also concluded that the only way was to change _UnalignedCheckpointCompatibilityITCase_ to wait for the job end. Thanks for the confirmation !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-758508420
> @rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ savepoint tests, I know we need to get the accumulators from _JobExecutionResult_ for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know Flink enough yet, can you give me a clue on this condition?
Hi @echauchot, from my point of view, there would be two possible semantics when users want to acquire accumulators:
1. Users want to retrieve the current value of accumulators immediately.
2. Users knows the job is bounded and want to acquire the final accumulator values after the job is finished.
Previously `getAccumulator()` takes the second semantics, and now we want to move it to the first semantics to make it consistent with the other client implementation. Therefore, if users still want to use the second semantics, I think they would need to call `getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)` explicitly. Thus for me to fix the failed tests we need to modify the tests since we would not be able to distinguish the first and second semantics automatically. What do you think about it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-756811406
@rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ I know we need to get the accumulators from _JobExecutionResult_ it that case for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know flink enough yet, can you give me a clue on this condition?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on a change in pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on a change in pull request #14558:
URL: https://github.com/apache/flink/pull/14558#discussion_r557317706
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniClusterJobClient.java
##########
@@ -106,7 +108,25 @@ public JobID getJobID() {
@Override
public CompletableFuture<Map<String, Object>> getAccumulators() {
- return getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults);
+ //TODO: this is not the only case when we need to get the accumulators from JobExecutionResult.
+ // It is needed also for UnalignedCheckpointCompatibilityITCase savepoints tests to pass.
+ // What is the complete condition ?
+ if (!miniCluster.isRunning()) {
Review comment:
agree `miniCluster.isRunning()` is more a side effect of `PerJobMiniClusterFactoryTest.testJobExecution` stopping the minicluster when the job is done. `jobResultFuture.isDone()` is definitely a better condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-768224928
@tillrohrmann can you please take a look at this PR as I saw you in the git history of this part of code ? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-760130203
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-755259773
@rmetzger I'm fixing the tests. But I have a flink starter question: how can I check if a given job is a streaming job ? Because if so I need to get the accumulators from job _ExecutionGraph_ otherwise from the _JobExecutionResult_
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-756811406
@rmetzger I fixed one of the failing tests. But for _UnalignedCheckpointCompatibilityITCase_ savepoint tests, I know we need to get the accumulators from _JobExecutionResult_ for the test to work. But I struggle to find the **proper condition** when to get the accumulators from _JobExecutionResult_ rather than getting them from the _ExecutionGraph_ as usual. I don't know Flink enough yet, can you give me a clue on this condition?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
}, {
"hash" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* c875ca96e006d84d17bf744cf2894954870ec194 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051)
* aa00e2a8a27c7ff2926b5dfa55f877a60fd4f4cf UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-768297169
@rmetzger don't worry, I know what it is :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754744583
R:@rmetzger. There are some test failures that I missed during my local tests. I'll ping you when this PR is ready for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
}, {
"hash" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11815",
"triggerID" : "5543d0dc0cc1d54619579ea2fdb855aac3085978",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760131077",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "760135401",
"triggerType" : "MANUAL"
}, {
"hash" : "c875ca96e006d84d17bf744cf2894954870ec194",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051",
"triggerID" : "c875ca96e006d84d17bf744cf2894954870ec194",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* c875ca96e006d84d17bf744cf2894954870ec194 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12051)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44801100ae245f2cd0d10d24830552b87164f072 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] echauchot removed a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
echauchot removed a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-755259773
@rmetzger I'm fixing the tests. But I have a flink starter question: how can I check if a given job is a streaming job ? Because if so I need to get the accumulators from job _ExecutionGraph_ otherwise from the _JobExecutionResult_
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14558:
URL: https://github.com/apache/flink/pull/14558#discussion_r555597955
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniClusterJobClient.java
##########
@@ -106,7 +108,25 @@ public JobID getJobID() {
@Override
public CompletableFuture<Map<String, Object>> getAccumulators() {
- return getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults);
+ //TODO: this is not the only case when we need to get the accumulators from JobExecutionResult.
+ // It is needed also for UnalignedCheckpointCompatibilityITCase savepoints tests to pass.
+ // What is the complete condition ?
+ if (!miniCluster.isRunning()) {
Review comment:
I'm wondering if the `miniCluster.isRunning()` is not a proper condition ? Since a mini-cluster could be viewed as a cluster that could run multiple jobs, the miniCluster would still be able to run after the job is done logically ? Perhaps we could use `jobResultFuture.isDone()`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniClusterJobClient.java
##########
@@ -106,7 +108,25 @@ public JobID getJobID() {
@Override
public CompletableFuture<Map<String, Object>> getAccumulators() {
- return getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults);
+ //TODO: this is not the only case when we need to get the accumulators from JobExecutionResult.
+ // It is needed also for UnalignedCheckpointCompatibilityITCase savepoints tests to pass.
+ // What is the complete condition ?
+ if (!miniCluster.isRunning()) {
Review comment:
I'm wondering if the `miniCluster.isRunning()` is not a proper condition ? Since a mini-cluster could be viewed as a cluster that could run multiple jobs, the miniCluster would still be able to run after the job is done logically ? Perhaps we could use `jobResultFuture.isDone()` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44801100ae245f2cd0d10d24830552b87164f072 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14558:
URL: https://github.com/apache/flink/pull/14558#issuecomment-754543918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "44801100ae245f2cd0d10d24830552b87164f072",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645",
"triggerID" : "44801100ae245f2cd0d10d24830552b87164f072",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44801100ae245f2cd0d10d24830552b87164f072 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11645)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org