You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/26 21:50:16 UTC
[GitHub] [flink] sjwiesman opened a new pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
sjwiesman opened a new pull request #13797:
URL: https://github.com/apache/flink/pull/13797
## What is the purpose of the change
This adds the new `CheckpointStorage` interface as described in FLIP-142 and wires it through the runtime. Public API changes will be added in a follow-up PR. This is based on #13794 and the first commit can be skipped.
## Brief changelog
6ab8c8d Adds the new `CheckpointStorage` interface, factory, and loader.
61a8ec6 Wires the `CheckpointStorage` object through the StreamGraph, ExecutionGraph, StreamTask, and CheckpointCoordinator
c06b3dc Removes the checkpoint-storage related methods from `StateBackend` and adds `CheckpointStorage` as an interface to the existing state backend implementations. This commit then makes misc fixes to tests so everything passes.
## Verifying this change
UT tests and also existing E2E tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-717232177
Fixed the missing license, sorry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 346e78df221bb7fae66979bed5d8e088c6f1c4e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433)
* f9196a71051202e88294076b7a514f895fb9f443 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 374508f59bd49283e374899444597ca761ce49c2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298)
* e6bf6df6365fc746d7054553b252a35c0a324a15 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563825336
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Agreed. I will open a follow-up ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
}, {
"hash" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392",
"triggerID" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"triggerType" : "PUSH"
}, {
"hash" : "38e62f33460561736e7a16e930e2d15376b6d095",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12399",
"triggerID" : "38e62f33460561736e7a16e930e2d15376b6d095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 9bf7f665540235030e0c428a7fd11cf6822074dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392)
* 38e62f33460561736e7a16e930e2d15376b6d095 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12399)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562165994
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageFactory.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+
+import java.io.IOException;
+
+/**
+ * A factory to create a specific {@link CheckpointStorage}. The storage creation gets a
+ * configuration object that can be used to read further config values.
+ *
+ * <p>The checkpoint storage factory is typically specified in the configuration to produce a
+ * configured storage backend.
+ *
+ * @param <T> The type of the checkpoint storage created.
+ */
+@PublicEvolving
+public interface CheckpointStorageFactory<T extends CheckpointStorage> {
+
+ /**
+ * Creates the checkpoint storage, optionally using the given configuration.
+ *
+ * @param config The Flink configuration (loaded by the TaskManager).
+ * @param classLoader The clsas loader that should be used to load the checkpoint storage.
+ * @return The created checkpoint storage.
+ * @throws IllegalConfigurationException If the configuration misses critical values, or
+ * specifies invalid values
+ * @throws IOException If the checkpoint storage initialization failed due to an I/O exception.
+ */
+ T createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException;
Review comment:
IIUC, this method only creates a "config" object and doesn't perform any IO.
If so it shouldn't throw `IOException`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/ConfigurableCheckpointStorage.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+
+/**
+ * An interface for checkpoint storage types that pick up additional parameters from a
+ * configuration.
+ */
+@Internal
+public interface ConfigurableCheckpointStorage {
Review comment:
IIUC, implementors are always `CheckpointStorage`.
If so it would be more clear to me if this interface extends `CheckpointStorage`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
Review comment:
Is this class missing `@PublicEvolving`?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
+
+ private final ClassLoader cl = getClass().getClassLoader();
+
+ @Test
+ public void testNoCheckpointStorageDefined() throws Exception {
+ assertNull(
+ CheckpointStorageLoader.loadCheckpointStorageFromConfig(
+ new Configuration(), cl, null));
+ }
+
+ @Test
+ public void testLegacyStateBackendTakesPrecedence() throws Exception {
+ StateBackend legacy = new LegacyStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Legacy state backends should always take precendence", legacy, configured);
+ }
+
+ @Test
+ public void testModernStateBackendDoesNotTakePrecedence() throws Exception {
+ StateBackend legacy = new ModernStateBackend();
Review comment:
Is it `modern` backend actually?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
+
+ private final ClassLoader cl = getClass().getClassLoader();
+
+ @Test
+ public void testNoCheckpointStorageDefined() throws Exception {
+ assertNull(
+ CheckpointStorageLoader.loadCheckpointStorageFromConfig(
+ new Configuration(), cl, null));
+ }
+
+ @Test
+ public void testLegacyStateBackendTakesPrecedence() throws Exception {
+ StateBackend legacy = new LegacyStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Legacy state backends should always take precendence", legacy, configured);
+ }
+
+ @Test
+ public void testModernStateBackendDoesNotTakePrecedence() throws Exception {
+ StateBackend legacy = new ModernStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Modern state backends should never take precendence", storage, configured);
+ }
+
+ @Test
+ public void testLoadingFromFactory() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, WorkingFactory.class.getName());
+ CheckpointStorage storage =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.assertThat(storage, Matchers.instanceOf(MockStorage.class));
+ }
+
+ @Test
+ public void testLoadingFails() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, "does.not.exist");
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a class that is not a factory
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, java.io.File.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a factory that fails
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, FailingFactory.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (IOException e) {
+ // expected
+ }
+ }
+
+ // A state backend that also implements checkpoint storage.
+ static final class LegacyStateBackend implements StateBackend, CheckpointStorage {
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class ModernStateBackend implements StateBackend {
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class MockStorage implements CheckpointStorage {
+
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+ }
+
+ static final class WorkingFactory implements CheckpointStorageFactory<MockStorage> {
+
+ @Override
+ public MockStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ return new MockStorage();
+ }
+ }
+
+ static final class FailingFactory implements CheckpointStorageFactory<CheckpointStorage> {
+
+ @Override
+ public CheckpointStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ throw new IOException("fail!");
Review comment:
The interface currently declares that it throws `IOException` (I asked a question about it).
If it stays then I'd expect that unchecked exception from here should be wrapped into IO (?).
If so, we should throw Runtime (or Test) exception from here to check that it get's wrapped.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
Review comment:
nit: rename to `CheckpointStorageLoad[er]Test`? (it's easier to find when names match)
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
Review comment:
I think the load order should be documented (javadoc). I'm assuming this is a part of public API.
nit: rename to just `load`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
Review comment:
Missing `storageName` looks like a misconfiguration. I'd expect `IllegalArgumentException` or a similar exception in this case.
If not, I think it would be less error-prone to return `Optional.empty`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
Review comment:
nit: `UnsupportedOperationException`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
Review comment:
nit: rename to just `fromConfig`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
Review comment:
How about `Optional` instead of `@Nullable`? (ditto logger in this and 2nd method).
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
+ StateBackend configuredStateBackend,
+ Configuration config,
+ ClassLoader classLoader,
+ @Nullable Logger logger)
+ throws IllegalConfigurationException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+ Preconditions.checkNotNull(configuredStateBackend, "statebackend");
+
+ // (1) Legacy state backends always take precedence
+ // for backwards compatibility.
+ if (configuredStateBackend instanceof CheckpointStorage) {
+ if (logger != null) {
+ logger.info(
+ "Using legacy state backend {} as Job checkpoint storage",
+ configuredStateBackend);
+ }
+
+ return (CheckpointStorage) configuredStateBackend;
+ }
+
+ CheckpointStorage storage;
+
+ // (2) Application defined checkpoint storage
+ if (fromApplication != null) {
+ // see if this is supposed to pick up additional configuration parameters
+ if (fromApplication instanceof ConfigurableCheckpointStorage) {
Review comment:
I think the branching in this method can be flattened; furthermore, method can be simplified by returning from each branch:
```
if (configuredStateBackend instanceof CheckpointStorage) {
return ...;
else if (fromApplication instanceof ConfigurableCheckpointStorage) {
return ...;
} else if (fromApplication != null) {
return ...;
} else {
return ...;
}
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
Review comment:
Does this field have to be public?
(ditto `FILE_SYSTEM_STORAGE_NAME`)
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
##########
@@ -468,12 +474,12 @@ private File getNextStoragePath() {
@Override
public CompletedCheckpointStorageLocation resolveCheckpoint(String pointer) throws IOException {
- return checkpointStreamBackend.resolveCheckpoint(pointer);
+ return ((CheckpointStorage) checkpointStreamBackend).resolveCheckpoint(pointer);
Review comment:
It seems possible to change the field type to `CheckpointStorage` and remove these casts.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -328,6 +329,7 @@ private void setBatchStateBackendAndTimerService(StreamGraph graph) {
if (useStateBackend) {
LOG.debug("Using BATCH execution state backend and timer service.");
graph.setStateBackend(new BatchExecutionStateBackend());
+ graph.setCheckpointStorage(new BatchExecutionCheckpointStorage());
Review comment:
Should there be a similar call for other cases (line 303)?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
I couldn't find rendering of this field in the UI.
Or is it implemented in the subsequent PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628)
* 486a8c8deda47645641acb285eb943d5d02cd13b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* f9196a71051202e88294076b7a514f895fb9f443 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451)
* 540d2dacd7426f974335e2690be6fa6fa7e7f818 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531)
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-766924751
Merged in 2426d84268ad330ec661a0ea6bedb2a3a2bd08e6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r512291906
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/ExecutorUtils.java
##########
@@ -79,6 +79,7 @@ public static void setBatchProperties(StreamGraph streamGraph, TableConfig table
streamGraph.setAllVerticesInSameSlotSharingGroupByDefault(false);
streamGraph.setScheduleMode(ScheduleMode.LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST);
streamGraph.setStateBackend(null);
+ streamGraph.setCheckpointStorage(null);
Review comment:
@aljoscha I'm pretty certain this is correct but could you please double-check. I'm also not sure if I need to explicitly do something for batch execution somewhere else now that it's supported for DataStream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman removed a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman removed a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-717232177
Fixed the missing license, sorry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562860032
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Forgot to backport, nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 346e78df221bb7fae66979bed5d8e088c6f1c4e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433)
* f9196a71051202e88294076b7a514f895fb9f443 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688)
* 06f933a05574758aa9c2472f7650b36f802e79cf UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562845447
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
+
+ private final ClassLoader cl = getClass().getClassLoader();
+
+ @Test
+ public void testNoCheckpointStorageDefined() throws Exception {
+ assertNull(
+ CheckpointStorageLoader.loadCheckpointStorageFromConfig(
+ new Configuration(), cl, null));
+ }
+
+ @Test
+ public void testLegacyStateBackendTakesPrecedence() throws Exception {
+ StateBackend legacy = new LegacyStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Legacy state backends should always take precendence", legacy, configured);
+ }
+
+ @Test
+ public void testModernStateBackendDoesNotTakePrecedence() throws Exception {
+ StateBackend legacy = new ModernStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Modern state backends should never take precendence", storage, configured);
+ }
+
+ @Test
+ public void testLoadingFromFactory() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, WorkingFactory.class.getName());
+ CheckpointStorage storage =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.assertThat(storage, Matchers.instanceOf(MockStorage.class));
+ }
+
+ @Test
+ public void testLoadingFails() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, "does.not.exist");
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a class that is not a factory
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, java.io.File.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a factory that fails
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, FailingFactory.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (IOException e) {
+ // expected
+ }
+ }
+
+ // A state backend that also implements checkpoint storage.
+ static final class LegacyStateBackend implements StateBackend, CheckpointStorage {
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class ModernStateBackend implements StateBackend {
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class MockStorage implements CheckpointStorage {
+
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+ }
+
+ static final class WorkingFactory implements CheckpointStorageFactory<MockStorage> {
+
+ @Override
+ public MockStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ return new MockStorage();
+ }
+ }
+
+ static final class FailingFactory implements CheckpointStorageFactory<CheckpointStorage> {
+
+ @Override
+ public CheckpointStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ throw new IOException("fail!");
Review comment:
I dropped the IOException. That's a leftover from an earlier iteration I missed when cleaning up the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
* 9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 4357aee8a598c554ba69ce22f38128c00ccef807 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412)
* 346e78df221bb7fae66979bed5d8e088c6f1c4e1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e6bf6df6365fc746d7054553b252a35c0a324a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302)
* e22fbccc6b2b9f0df0d5308974eabc5c30d22975 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563694590
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -328,6 +329,7 @@ private void setBatchStateBackendAndTimerService(StreamGraph graph) {
if (useStateBackend) {
LOG.debug("Using BATCH execution state backend and timer service.");
graph.setStateBackend(new BatchExecutionStateBackend());
+ graph.setCheckpointStorage(new BatchExecutionCheckpointStorage());
Review comment:
Got it. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562862511
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -328,6 +329,7 @@ private void setBatchStateBackendAndTimerService(StreamGraph graph) {
if (useStateBackend) {
LOG.debug("Using BATCH execution state backend and timer service.");
graph.setStateBackend(new BatchExecutionStateBackend());
+ graph.setCheckpointStorage(new BatchExecutionCheckpointStorage());
Review comment:
This was in the follow up PR. This is the weird space where we are getting close the public API. I have backported `graph.setCheckpointStorage` on line 303ish but not setting the instance variable `checkpointStorage` that I have left for the follow up because it involves modifying `StreamExecutionEnvironment`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* f9196a71051202e88294076b7a514f895fb9f443 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e59969cd8feed88248332fd4c2b63ff873bb093a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296)
* 374508f59bd49283e374899444597ca761ce49c2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716844372
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit c06b3dcd4eb4045f66352c8b65c14a7b60163ecd (Mon Oct 26 21:52:17 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e6bf6df6365fc746d7054553b252a35c0a324a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563825336
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Agreed. I will open a follow-up ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562856816
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
Review comment:
This is required to maintain proper backward compatibility with existing flink-confs. While we cannot make this config strictly required I will add some warning level logging encouraging users to add it.
Also, +1 on the Optional.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e59969cd8feed88248332fd4c2b63ff873bb093a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563688434
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
Review comment:
Do you mean
> Do not use Optional as a function argument, instead either overload the method or use the Builder pattern for the set of function arguments.
I think the intention is to avoid arguments that can be empty/null. According to it the method should be refactored.
But I think the very purpose of this method is to chose which arguments to use. And for that I'd use `Optional`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 5cb0bef03180f5939c67cb074bbbcc5fd17105b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r512291906
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/ExecutorUtils.java
##########
@@ -79,6 +79,7 @@ public static void setBatchProperties(StreamGraph streamGraph, TableConfig table
streamGraph.setAllVerticesInSameSlotSharingGroupByDefault(false);
streamGraph.setScheduleMode(ScheduleMode.LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST);
streamGraph.setStateBackend(null);
+ streamGraph.setCheckpointStorage(null);
Review comment:
@aljoscha I'm pretty certain this is correct but could you please double-check. I'm also not sure if I need to explicitly do something for batch execution somewhere else now that it's supported for DataStream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
* 9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 540d2dacd7426f974335e2690be6fa6fa7e7f818 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531)
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562842079
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
Review comment:
Flink's style guide says not to use `Optional` for function arguments.
https://flink.apache.org/contributing/code-style-and-quality-java.html#java-optional
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 486a8c8deda47645641acb285eb943d5d02cd13b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 5cb0bef03180f5939c67cb074bbbcc5fd17105b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330)
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
}, {
"hash" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392",
"triggerID" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"triggerType" : "PUSH"
}, {
"hash" : "38e62f33460561736e7a16e930e2d15376b6d095",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12399",
"triggerID" : "38e62f33460561736e7a16e930e2d15376b6d095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 38e62f33460561736e7a16e930e2d15376b6d095 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12399)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562866385
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
I didn't add it to the UI because the state_backend field is also not rendered in the UI. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman closed pull request #13797:
URL: https://github.com/apache/flink/pull/13797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562848702
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
##########
@@ -468,12 +474,12 @@ private File getNextStoragePath() {
@Override
public CompletedCheckpointStorageLocation resolveCheckpoint(String pointer) throws IOException {
- return checkpointStreamBackend.resolveCheckpoint(pointer);
+ return ((CheckpointStorage) checkpointStreamBackend).resolveCheckpoint(pointer);
Review comment:
I would prefer not to, there are then a number of places where I would need to cast it back to StateBackend, so there will be casts. Or I maintain the reference in two variables and that seems more error-prone. RocksDBStateBackend is deprecated in the follow-up PR anyway so I don't think this is that problematic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
}, {
"hash" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e22fbccc6b2b9f0df0d5308974eabc5c30d22975 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346)
* 9bf7f665540235030e0c428a7fd11cf6822074dc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-767315581
Hoops, I commented before I noticed this PR is closed.
There are a couple of places that "state-backed" should be replaced by "checkpoint storage", but we can fix them in the follow-up PRs I guess.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-765647307
@rkhachatryan appreciate the speedy review. I hope I've addressed all your comments. Please take a look and let me know if you have any other concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562165994
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageFactory.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+
+import java.io.IOException;
+
+/**
+ * A factory to create a specific {@link CheckpointStorage}. The storage creation gets a
+ * configuration object that can be used to read further config values.
+ *
+ * <p>The checkpoint storage factory is typically specified in the configuration to produce a
+ * configured storage backend.
+ *
+ * @param <T> The type of the checkpoint storage created.
+ */
+@PublicEvolving
+public interface CheckpointStorageFactory<T extends CheckpointStorage> {
+
+ /**
+ * Creates the checkpoint storage, optionally using the given configuration.
+ *
+ * @param config The Flink configuration (loaded by the TaskManager).
+ * @param classLoader The clsas loader that should be used to load the checkpoint storage.
+ * @return The created checkpoint storage.
+ * @throws IllegalConfigurationException If the configuration misses critical values, or
+ * specifies invalid values
+ * @throws IOException If the checkpoint storage initialization failed due to an I/O exception.
+ */
+ T createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException;
Review comment:
IIUC, this method only creates a "config" object and doesn't perform any IO.
If so it shouldn't throw `IOException`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/ConfigurableCheckpointStorage.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+
+/**
+ * An interface for checkpoint storage types that pick up additional parameters from a
+ * configuration.
+ */
+@Internal
+public interface ConfigurableCheckpointStorage {
Review comment:
IIUC, implementors are always `CheckpointStorage`.
If so it would be more clear to me if this interface extends `CheckpointStorage`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
Review comment:
Is this class missing `@PublicEvolving`?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
+
+ private final ClassLoader cl = getClass().getClassLoader();
+
+ @Test
+ public void testNoCheckpointStorageDefined() throws Exception {
+ assertNull(
+ CheckpointStorageLoader.loadCheckpointStorageFromConfig(
+ new Configuration(), cl, null));
+ }
+
+ @Test
+ public void testLegacyStateBackendTakesPrecedence() throws Exception {
+ StateBackend legacy = new LegacyStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Legacy state backends should always take precendence", legacy, configured);
+ }
+
+ @Test
+ public void testModernStateBackendDoesNotTakePrecedence() throws Exception {
+ StateBackend legacy = new ModernStateBackend();
Review comment:
Is it `modern` backend actually?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
+
+ private final ClassLoader cl = getClass().getClassLoader();
+
+ @Test
+ public void testNoCheckpointStorageDefined() throws Exception {
+ assertNull(
+ CheckpointStorageLoader.loadCheckpointStorageFromConfig(
+ new Configuration(), cl, null));
+ }
+
+ @Test
+ public void testLegacyStateBackendTakesPrecedence() throws Exception {
+ StateBackend legacy = new LegacyStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Legacy state backends should always take precendence", legacy, configured);
+ }
+
+ @Test
+ public void testModernStateBackendDoesNotTakePrecedence() throws Exception {
+ StateBackend legacy = new ModernStateBackend();
+ CheckpointStorage storage = new MockStorage();
+
+ CheckpointStorage configured =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ storage, legacy, new Configuration(), cl, null);
+
+ Assert.assertEquals(
+ "Modern state backends should never take precendence", storage, configured);
+ }
+
+ @Test
+ public void testLoadingFromFactory() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, WorkingFactory.class.getName());
+ CheckpointStorage storage =
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.assertThat(storage, Matchers.instanceOf(MockStorage.class));
+ }
+
+ @Test
+ public void testLoadingFails() throws Exception {
+ final Configuration config = new Configuration();
+
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, "does.not.exist");
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a class that is not a factory
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, java.io.File.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (DynamicCodeLoadingException e) {
+ // expected
+ }
+
+ // try a factory that fails
+ config.setString(CheckpointingOptions.CHECKPOINT_STORAGE, FailingFactory.class.getName());
+ try {
+ CheckpointStorageLoader.fromApplicationOrConfigOrDefault(
+ null, new ModernStateBackend(), config, cl, null);
+ Assert.fail("should fail with exception");
+ } catch (IOException e) {
+ // expected
+ }
+ }
+
+ // A state backend that also implements checkpoint storage.
+ static final class LegacyStateBackend implements StateBackend, CheckpointStorage {
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class ModernStateBackend implements StateBackend {
+
+ @Override
+ public <K> CheckpointableKeyedStateBackend<K> createKeyedStateBackend(
+ Environment env,
+ JobID jobID,
+ String operatorIdentifier,
+ TypeSerializer<K> keySerializer,
+ int numberOfKeyGroups,
+ KeyGroupRange keyGroupRange,
+ TaskKvStateRegistry kvStateRegistry,
+ TtlTimeProvider ttlTimeProvider,
+ MetricGroup metricGroup,
+ Collection<KeyedStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+
+ @Override
+ public OperatorStateBackend createOperatorStateBackend(
+ Environment env,
+ String operatorIdentifier,
+ Collection<OperatorStateHandle> stateHandles,
+ CloseableRegistry cancelStreamRegistry)
+ throws Exception {
+ return null;
+ }
+ }
+
+ static final class MockStorage implements CheckpointStorage {
+
+ @Override
+ public CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {
+ return null;
+ }
+ }
+
+ static final class WorkingFactory implements CheckpointStorageFactory<MockStorage> {
+
+ @Override
+ public MockStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ return new MockStorage();
+ }
+ }
+
+ static final class FailingFactory implements CheckpointStorageFactory<CheckpointStorage> {
+
+ @Override
+ public CheckpointStorage createFromConfig(ReadableConfig config, ClassLoader classLoader)
+ throws IllegalConfigurationException, IOException {
+ throw new IOException("fail!");
Review comment:
The interface currently declares that it throws `IOException` (I asked a question about it).
If it stays then I'd expect that unchecked exception from here should be wrapped into IO (?).
If so, we should throw Runtime (or Test) exception from here to check that it get's wrapped.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStorageLoadingTest.java
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.query.TaskKvStateRegistry;
+import org.apache.flink.runtime.state.ttl.TtlTimeProvider;
+import org.apache.flink.util.DynamicCodeLoadingException;
+
+import org.hamcrest.Matchers;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import static org.junit.Assert.assertNull;
+
+/** This test validates that checkpoint storage is properly loaded from configuration. */
+public class CheckpointStorageLoadingTest {
Review comment:
nit: rename to `CheckpointStorageLoad[er]Test`? (it's easier to find when names match)
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
Review comment:
I think the load order should be documented (javadoc). I'm assuming this is a part of public API.
nit: rename to just `load`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
Review comment:
Missing `storageName` looks like a misconfiguration. I'd expect `IllegalArgumentException` or a similar exception in this case.
If not, I think it would be less error-prone to return `Optional.empty`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
Review comment:
nit: `UnsupportedOperationException`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
Review comment:
nit: rename to just `fromConfig`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
Review comment:
How about `Optional` instead of `@Nullable`? (ditto logger in this and 2nd method).
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
+ StateBackend configuredStateBackend,
+ Configuration config,
+ ClassLoader classLoader,
+ @Nullable Logger logger)
+ throws IllegalConfigurationException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+ Preconditions.checkNotNull(configuredStateBackend, "statebackend");
+
+ // (1) Legacy state backends always take precedence
+ // for backwards compatibility.
+ if (configuredStateBackend instanceof CheckpointStorage) {
+ if (logger != null) {
+ logger.info(
+ "Using legacy state backend {} as Job checkpoint storage",
+ configuredStateBackend);
+ }
+
+ return (CheckpointStorage) configuredStateBackend;
+ }
+
+ CheckpointStorage storage;
+
+ // (2) Application defined checkpoint storage
+ if (fromApplication != null) {
+ // see if this is supposed to pick up additional configuration parameters
+ if (fromApplication instanceof ConfigurableCheckpointStorage) {
Review comment:
I think the branching in this method can be flattened; furthermore, method can be simplified by returning from each branch:
```
if (configuredStateBackend instanceof CheckpointStorage) {
return ...;
else if (fromApplication instanceof ConfigurableCheckpointStorage) {
return ...;
} else if (fromApplication != null) {
return ...;
} else {
return ...;
}
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
Review comment:
Does this field have to be public?
(ditto `FILE_SYSTEM_STORAGE_NAME`)
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
##########
@@ -468,12 +474,12 @@ private File getNextStoragePath() {
@Override
public CompletedCheckpointStorageLocation resolveCheckpoint(String pointer) throws IOException {
- return checkpointStreamBackend.resolveCheckpoint(pointer);
+ return ((CheckpointStorage) checkpointStreamBackend).resolveCheckpoint(pointer);
Review comment:
It seems possible to change the field type to `CheckpointStorage` and remove these casts.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -328,6 +329,7 @@ private void setBatchStateBackendAndTimerService(StreamGraph graph) {
if (useStateBackend) {
LOG.debug("Using BATCH execution state backend and timer service.");
graph.setStateBackend(new BatchExecutionStateBackend());
+ graph.setCheckpointStorage(new BatchExecutionCheckpointStorage());
Review comment:
Should there be a similar call for other cases (line 303)?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
I couldn't find rendering of this field in the UI.
Or is it implemented in the subsequent PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624)
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
* 9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624)
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* f9196a71051202e88294076b7a514f895fb9f443 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451)
* 540d2dacd7426f974335e2690be6fa6fa7e7f818 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 540d2dacd7426f974335e2690be6fa6fa7e7f818 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531)
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628)
* 486a8c8deda47645641acb285eb943d5d02cd13b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 06f933a05574758aa9c2472f7650b36f802e79cf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292)
* e59969cd8feed88248332fd4c2b63ff873bb093a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e6bf6df6365fc746d7054553b252a35c0a324a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302)
* e22fbccc6b2b9f0df0d5308974eabc5c30d22975 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 942af8415dd1a9fbd715532bf6b578a7a4b7f082 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543)
* 9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624)
* 550bb034ad8fecc1cfc42840ff17464eac1c04e6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 5cb0bef03180f5939c67cb074bbbcc5fd17105b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330)
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 4357aee8a598c554ba69ce22f38128c00ccef807 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e59969cd8feed88248332fd4c2b63ff873bb093a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296)
* 374508f59bd49283e374899444597ca761ce49c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-766908446
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563688434
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
+ e);
+ } catch (ClassCastException | InstantiationException | IllegalAccessException e) {
+ throw new DynamicCodeLoadingException(
+ "The class configured under '"
+ + CheckpointingOptions.CHECKPOINT_STORAGE.key()
+ + "' is not a valid checkpoint storage factory ("
+ + storageName
+ + ')',
+ e);
+ }
+
+ return factory.createFromConfig(config, classLoader);
+ }
+ }
+
+ public static CheckpointStorage fromApplicationOrConfigOrDefault(
+ @Nullable CheckpointStorage fromApplication,
Review comment:
Do you mean
> Do not use Optional as a function argument, instead either overload the method or use the Builder pattern for the set of function arguments.
I think the intention is to avoid arguments that can be empty/null. According to it the method should be refactored.
But I think the very purpose of this method is to chose which arguments to use. And for that I'd use `Optional`.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
##########
@@ -328,6 +329,7 @@ private void setBatchStateBackendAndTimerService(StreamGraph graph) {
if (useStateBackend) {
LOG.debug("Using BATCH execution state backend and timer service.");
graph.setStateBackend(new BatchExecutionStateBackend());
+ graph.setCheckpointStorage(new BatchExecutionCheckpointStorage());
Review comment:
Got it. Thanks.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Hmm..you're right. I think it would be nice to show it but can be implemented in a separate PR/ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* e22fbccc6b2b9f0df0d5308974eabc5c30d22975 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716844091
Note to reviewers, I decided not update the javadoc for StateBackend in this PR. I'll update it in the follow up when I add the new StateBackend implementations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 374508f59bd49283e374899444597ca761ce49c2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298)
* e6bf6df6365fc746d7054553b252a35c0a324a15 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563700678
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Hmm..you're right. I think it would be nice to show it but can be implemented in a separate PR/ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
}, {
"hash" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392",
"triggerID" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"triggerType" : "PUSH"
}, {
"hash" : "38e62f33460561736e7a16e930e2d15376b6d095",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38e62f33460561736e7a16e930e2d15376b6d095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 9bf7f665540235030e0c428a7fd11cf6822074dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392)
* 38e62f33460561736e7a16e930e2d15376b6d095 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
}, {
"hash" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12296",
"triggerID" : "e59969cd8feed88248332fd4c2b63ff873bb093a",
"triggerType" : "PUSH"
}, {
"hash" : "374508f59bd49283e374899444597ca761ce49c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12298",
"triggerID" : "374508f59bd49283e374899444597ca761ce49c2",
"triggerType" : "PUSH"
}, {
"hash" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12302",
"triggerID" : "e6bf6df6365fc746d7054553b252a35c0a324a15",
"triggerType" : "PUSH"
}, {
"hash" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12346",
"triggerID" : "e22fbccc6b2b9f0df0d5308974eabc5c30d22975",
"triggerType" : "PUSH"
}, {
"hash" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392",
"triggerID" : "9bf7f665540235030e0c428a7fd11cf6822074dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 9bf7f665540235030e0c428a7fd11cf6822074dc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12392)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
}, {
"hash" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8543",
"triggerID" : "942af8415dd1a9fbd715532bf6b578a7a4b7f082",
"triggerType" : "PUSH"
}, {
"hash" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8624",
"triggerID" : "9e0077b6b164ff27cf2ef226f0c26601e9f1b0ea",
"triggerType" : "PUSH"
}, {
"hash" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8628",
"triggerID" : "550bb034ad8fecc1cfc42840ff17464eac1c04e6",
"triggerType" : "PUSH"
}, {
"hash" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8641",
"triggerID" : "486a8c8deda47645641acb285eb943d5d02cd13b",
"triggerType" : "PUSH"
}, {
"hash" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8688",
"triggerID" : "1cba28c6f81ac8ec4fbea9b6ffd237d346165ebd",
"triggerType" : "PUSH"
}, {
"hash" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292",
"triggerID" : "06f933a05574758aa9c2472f7650b36f802e79cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 06f933a05574758aa9c2472f7650b36f802e79cf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12292)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562841036
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
Review comment:
It should actually be internal, nice catch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 346e78df221bb7fae66979bed5d8e088c6f1c4e1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 4357aee8a598c554ba69ce22f38128c00ccef807 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412)
* 346e78df221bb7fae66979bed5d8e088c6f1c4e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-766908446
Thanks! Will merge and rebase the follow-up later today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
}, {
"hash" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8433",
"triggerID" : "346e78df221bb7fae66979bed5d8e088c6f1c4e1",
"triggerType" : "PUSH"
}, {
"hash" : "f9196a71051202e88294076b7a514f895fb9f443",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451",
"triggerID" : "f9196a71051202e88294076b7a514f895fb9f443",
"triggerType" : "PUSH"
}, {
"hash" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531",
"triggerID" : "540d2dacd7426f974335e2690be6fa6fa7e7f818",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* f9196a71051202e88294076b7a514f895fb9f443 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8451)
* 540d2dacd7426f974335e2690be6fa6fa7e7f818 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716844372
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 38e62f33460561736e7a16e930e2d15376b6d095 (Fri May 28 08:06:26 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman closed pull request #13797:
URL: https://github.com/apache/flink/pull/13797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r563782279
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -34,6 +34,16 @@
.noDefaultValue()
.withDescription("The state backend to be used to store and checkpoint state.");
+ /** The checkpoint storage used to checkpoint state. */
+ @Documentation.Section(value = Documentation.Sections.COMMON_STATE_BACKENDS, position = 2)
+ @Documentation.ExcludeFromDocumentation(
+ "Hidden until FileSystemStorage and JobManagerStorage are implemented")
+ public static final ConfigOption<String> CHECKPOINT_STORAGE =
+ ConfigOptions.key("state.checkpoint-storage")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The state backend to be used to checkpoint state.");
Review comment:
"The state backend to be used to checkpoint state." ==>
"The checkpoint storage to be used to store checkpoints"
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorage.java
##########
@@ -0,0 +1,67 @@
+/*
Review comment:
Maybe add a "see also" reference of `CheckpointStorage` in the Java doc of `StateBackend`? The `StateBackend` Java Doc is the same as before which mentions "durable persistent storage".
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java
##########
@@ -231,15 +233,15 @@ private static void throwNonRestoredStateException(
// ------------------------------------------------------------------------
public static void disposeSavepoint(
- String pointer, StateBackend stateBackend, ClassLoader classLoader)
+ String pointer, CheckpointStorage checkpointStorage, ClassLoader classLoader)
throws IOException, FlinkException {
checkNotNull(pointer, "location");
- checkNotNull(stateBackend, "stateBackend");
+ checkNotNull(checkpointStorage, "stateBackend");
Review comment:
"stateBackend" => "checkpoint storage"
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -58,28 +60,35 @@
* the factory class was not found or the factory could not be instantiated
* @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
* creating / configuring the checkpoint storage in the factory
- * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
- * checkpoint storage
*/
- public static CheckpointStorage loadCheckpointStorageFromConfig(
+ public static Optional<CheckpointStorage> fromConfig(
ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
- throws IllegalStateException, DynamicCodeLoadingException, IOException {
+ throws IllegalStateException, DynamicCodeLoadingException {
Preconditions.checkNotNull(config, "config");
Preconditions.checkNotNull(classLoader, "classLoader");
final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
if (storageName == null) {
- return null;
+ if (logger != null) {
+ logger.warn(
+ "The configuration {} has not be set in the current"
+ + " sessions flink-conf.yaml. Falling back to a default CheckpointStorage"
+ + " type. Users are strongly encouraged explicitly set this configuration"
Review comment:
"Users are strongly encouraged explicitly" =>
Users are strongly encouraged to explicitly
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
+ }
+
+ CheckpointStorageFactory<?> factory;
+ try {
+ @SuppressWarnings("rawtypes")
+ Class<? extends CheckpointStorageFactory> clazz =
+ Class.forName(storageName, false, classLoader)
+ .asSubclass(CheckpointStorageFactory.class);
+
+ factory = clazz.newInstance();
+ } catch (ClassNotFoundException e) {
+ throw new DynamicCodeLoadingException(
+ "Cannot find configured state backend factory class: " + storageName,
Review comment:
"state backend" => "checkpoint storage"
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
+ * of a {@link CheckpointStorageFactory}. If a CheckpointStorageFactory class name is specified,
+ * the factory is instantiated (via its zero-argument constructor) and its {@link
+ * CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)} method is called.
+ *
+ * <p>Recognized shortcut names are '{@value #JOB_MANAGER_STORAGE_NAME}', and '{@value
+ * #FILE_SYSTEM_STORAGE_NAME}'.
+ *
+ * @param config The configuration to load the checkpoint storage from
+ * @param classLoader The class loader that should be used to load the checkpoint storage
+ * @param logger Optionally, a logger to log actions to (may be null)
+ * @return The instantiated checkpoint storage.
+ * @throws DynamicCodeLoadingException Thrown if a checkpoint storage factory is configured and
+ * the factory class was not found or the factory could not be instantiated
+ * @throws IllegalConfigurationException May be thrown by the CheckpointStorageFactory when
+ * creating / configuring the checkpoint storage in the factory
+ * @throws IOException May be thrown by the CheckpointStorageFactory when instantiating the
+ * checkpoint storage
+ */
+ public static CheckpointStorage loadCheckpointStorageFromConfig(
+ ReadableConfig config, ClassLoader classLoader, @Nullable Logger logger)
+ throws IllegalStateException, DynamicCodeLoadingException, IOException {
+
+ Preconditions.checkNotNull(config, "config");
+ Preconditions.checkNotNull(classLoader, "classLoader");
+
+ final String storageName = config.get(CheckpointingOptions.CHECKPOINT_STORAGE);
+ if (storageName == null) {
+ return null;
+ }
+
+ switch (storageName.toLowerCase()) {
+ case JOB_MANAGER_STORAGE_NAME:
+ throw new IllegalStateException(
+ "JobManagerCheckpointStorage is not yet implemented");
+
+ case FILE_SYSTEM_STORAGE_NAME:
+ throw new IllegalStateException(
+ "FileSystemCheckpointStorage is not yet implemented");
+
+ default:
+ if (logger != null) {
+ logger.info("Loading state backend via factory {}", storageName);
Review comment:
"state backend" => "checkpoint storage"
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.util.DynamicCodeLoadingException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/** This class contains utility methods to load checkpoint storage from configurations. */
+public class CheckpointStorageLoader {
+
+ public static final String JOB_MANAGER_STORAGE_NAME = "jobmanager";
+
+ public static final String FILE_SYSTEM_STORAGE_NAME = "filesystem";
+
+ /**
+ * Loads the checkpoint storage from the configuration, from the parameter
+ * 'state.checkpoint-storage', as defined in {@link CheckpointingOptions#CHECKPOINT_STORAGE}.
+ *
+ * <p>The state backends can be specified either via their shortcut name, or via the class name
Review comment:
"The state backends can be specified either via their shortcut name" =>
"The checkpoint storage can be specified either via its shortcut name"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8330",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
}, {
"hash" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185e46fd046c5f39a87ac3583ad559b065a275e1",
"triggerType" : "PUSH"
}, {
"hash" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412",
"triggerID" : "4357aee8a598c554ba69ce22f38128c00ccef807",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 185e46fd046c5f39a87ac3583ad559b065a275e1 UNKNOWN
* 4357aee8a598c554ba69ce22f38128c00ccef807 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8412)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13797:
URL: https://github.com/apache/flink/pull/13797#issuecomment-716854695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c06b3dcd4eb4045f66352c8b65c14a7b60163ecd",
"triggerType" : "PUSH"
}, {
"hash" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5cb0bef03180f5939c67cb074bbbcc5fd17105b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c06b3dcd4eb4045f66352c8b65c14a7b60163ecd UNKNOWN
* 5cb0bef03180f5939c67cb074bbbcc5fd17105b8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #13797: [FLINK-19465][runtime / statebackends] Add CheckpointStorage interface and wire through runtime
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #13797:
URL: https://github.com/apache/flink/pull/13797#discussion_r562860032
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java
##########
@@ -79,6 +81,9 @@
@JsonProperty(FIELD_NAME_STATE_BACKEND)
private final String stateBackend;
+ @JsonProperty(FIELD_NAME_CHECKPOINT_STORAGE)
+ private final String checkpointStorage;
Review comment:
Forgot to backport, nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org