You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/20 16:10:39 UTC
[GitHub] [flink] rkhachatryan opened a new pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
rkhachatryan opened a new pull request #13709:
URL: https://github.com/apache/flink/pull/13709
## What is the purpose of the change
If checkpoints in ZK are the same as already downloaded then skip
download. This prevents excessive S3 reads and eventually failure to
start the job, particularly in multi-region recovery.
## Verifying this change
Added `testNoDownloadIfCheckpointsNotChanged` and `testDownloadIfCheckpointsChanged` to `ZooKeeperCompletedCheckpointStoreTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712965507
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1fa8bf62c353d0b3e087826271476529a5e4622a (Tue Oct 20 16:13:49 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19401).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-713839243
Thanks for reviewing and merging!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 6af74f96aa6de790a7d28afd6d438ed8959da4be UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-713453964
There was a failure of `ZooKeeperHighAvailabilityITCase` after my changes.
If I understood correctly, it tests whether failure to retrieve a checkpoint fails recovery. This assumption doesn't hold now (if the checkpoints are already in memory in ids match ones in ZK). The test was added in FLINK-8487.
I couldn't find a way to re-create ZKStore to force retrieval -> to raise error -> to pass the test.
So I just replaced ITCase with a unit test to check whether failure of `handle.retrieve` causes failure of `ZKStore.recover()` .
@aljoscha, can you take a look?
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise merged pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #13709:
URL: https://github.com/apache/flink/pull/13709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13709:
URL: https://github.com/apache/flink/pull/13709#discussion_r509608347
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java
##########
@@ -143,6 +145,10 @@ public void recover() throws Exception {
int numberOfInitialCheckpoints = initialCheckpoints.size();
LOG.info("Found {} checkpoints in ZooKeeper.", numberOfInitialCheckpoints);
+ if (haveAllDownloaded(initialCheckpoints)) {
Review comment:
I was thinking of optimization, but I understand your argument.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa8bf62c353d0b3e087826271476529a5e4622a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960)
* 14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13709:
URL: https://github.com/apache/flink/pull/13709#discussion_r509211683
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java
##########
@@ -143,6 +145,10 @@ public void recover() throws Exception {
int numberOfInitialCheckpoints = initialCheckpoints.size();
LOG.info("Found {} checkpoints in ZooKeeper.", numberOfInitialCheckpoints);
+ if (haveAllDownloaded(initialCheckpoints)) {
Review comment:
What happens if all but one checkpoint have been downloaded? Would it download all of them or just the missing one?
If the former, I'd modify this method to return all missing checkpoints instead.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStoreTest.java
##########
@@ -58,6 +66,54 @@ public void testPathConversion() {
assertEquals(checkpointId, ZooKeeperCompletedCheckpointStore.pathToCheckpointId(path));
}
+ private static class TestException extends RuntimeException{
Review comment:
Could we reuse `ExpectedTestException` from `runtime` `testutils` package?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"triggerType" : "PUSH"
}, {
"hash" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8023",
"triggerID" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6af74f96aa6de790a7d28afd6d438ed8959da4be Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001)
* 3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8023)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13709:
URL: https://github.com/apache/flink/pull/13709#discussion_r509265695
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java
##########
@@ -143,6 +145,10 @@ public void recover() throws Exception {
int numberOfInitialCheckpoints = initialCheckpoints.size();
LOG.info("Found {} checkpoints in ZooKeeper.", numberOfInitialCheckpoints);
+ if (haveAllDownloaded(initialCheckpoints)) {
Review comment:
Yes, if at least one checkpoint is missing (or some extra checkpoints exist) it will re-download `initialCheckpoints`.
It's quite a sensitive part affecting correctness so I think it's better to keep it as simple as possible and not optimize further.
With that in mind, I think the only use of returning missing checkpoints would be to log them.
Is this what you propose?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa8bf62c353d0b3e087826271476529a5e4622a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960)
* 14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"triggerType" : "PUSH"
}, {
"hash" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8023",
"triggerID" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8023)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa8bf62c353d0b3e087826271476529a5e4622a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-713801456
CI failure isn't related to PR changes, private build succeeded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 6af74f96aa6de790a7d28afd6d438ed8959da4be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fa8bf62c353d0b3e087826271476529a5e4622a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-712988228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7960",
"triggerID" : "1fa8bf62c353d0b3e087826271476529a5e4622a",
"triggerType" : "PUSH"
}, {
"hash" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7965",
"triggerID" : "14f7ba792a5ff15a1e6c2ca8cae93c7f3c63fe7c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "713453964",
"triggerType" : "MANUAL"
}, {
"hash" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001",
"triggerID" : "6af74f96aa6de790a7d28afd6d438ed8959da4be",
"triggerType" : "PUSH"
}, {
"hash" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 6af74f96aa6de790a7d28afd6d438ed8959da4be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8001)
* 3dd7b648660b9c4ae7fdb2bcff5f19bd02caba5c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13709:
URL: https://github.com/apache/flink/pull/13709#discussion_r509267021
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStoreTest.java
##########
@@ -58,6 +66,54 @@ public void testPathConversion() {
assertEquals(checkpointId, ZooKeeperCompletedCheckpointStore.pathToCheckpointId(path));
}
+ private static class TestException extends RuntimeException{
Review comment:
I searched for suitable `TestException` and missed `ExpectedTestException` :)
Will use it, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Jiayi-Liao commented on a change in pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
Jiayi-Liao commented on a change in pull request #13709:
URL: https://github.com/apache/flink/pull/13709#discussion_r509010157
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java
##########
@@ -143,6 +145,10 @@ public void recover() throws Exception {
int numberOfInitialCheckpoints = initialCheckpoints.size();
LOG.info("Found {} checkpoints in ZooKeeper.", numberOfInitialCheckpoints);
+ if (haveAllDownloaded(initialCheckpoints)) {
+ LOG.info("All checkpoints found are already downloaded.");
Review comment:
Also log the count of "All checkpoints" ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #13709: [FLINK-19401][checkpointing] Download checkpoints only if needed
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #13709:
URL: https://github.com/apache/flink/pull/13709#issuecomment-713801456
CI failure not related, private build succeeded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org