You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:53:13 UTC

[GitHub] [flink] wangyang0918 opened a new pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

wangyang0918 opened a new pull request #13871:
URL: https://github.com/apache/flink/pull/13871


   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   This is second PR of [FLIP-144](https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink#FLIP144:NativeKubernetesHAforFlink-Resourceversion), which will introduces `KubernetesCheckpointRecoveryFactory`. It is responsible for creating the `CompletedCheckpointStore` and `CheckpointIDCounter` based on Kubernetes API.
   
   This PR is based on #13864, Please start review with the commit with title **_[hotfix][test] Make TestingRetrievableStateStorageHelper is settable with exception_**.
   
   Each job will have a dedicated ConfigMap, which is the JobManager leader, for storing the checkpoint pointers. It is just like following.
   
   ```
   wangyang-pc:apache-flink-k8s-ha danrtsey.wy$ kubectl get cm k8s-ha-app-pr3-00000000000000000000000000000000-jobmanager-leader -o yaml
   apiVersion: v1
   data:
     address: akka.tcp://flink@172.20.1.17:6123/user/rpc/jobmanager_2
     checkpointID-0000000000000039144: rO0ABXNyADtvcmcuYXBhY2hlLmZsaW5rLnJ1bnRpbWUuc3RhdGUuUmV0cmlldmFibGVTdHJlYW1TdGF0ZUhhbmRsZQABHhjxVZcrAgABTAAYd3JhcHBlZFN0cmVhbVN0YXRlSGFuZGxldAAyTG9yZy9hcGFjaGUvZmxpbmsvcnVudGltZS9zdGF0ZS9TdHJlYW1TdGF0ZUhhbmRsZTt4cHNyADlvcmcuYXBhY2hlLmZsaW5rLnJ1bnRpbWUuc3RhdGUuZmlsZXN5c3RlbS5GaWxlU3RhdGVIYW5kbGUE3HXYYr0bswIAAkoACXN0YXRlU2l6ZUwACGZpbGVQYXRodAAfTG9yZy9hcGFjaGUvZmxpbmsvY29yZS9mcy9QYXRoO3hwAAAAAAACSkNzcgAdb3JnLmFwYWNoZS5mbGluay5jb3JlLmZzLlBhdGgAAAAAAAAAAQIAAUwAA3VyaXQADkxqYXZhL25ldC9VUkk7eHBzcgAMamF2YS5uZXQuVVJJrAF4LkOeSasDAAFMAAZzdHJpbmd0ABJMamF2YS9sYW5nL1N0cmluZzt4cHQATm9zczovL2ZsaW5rLWRlYnVnLXlpcWkvZmxpbmstaGEvazhzLWhhLWFwcC1wcjMvY29tcGxldGVkQ2hlY2twb2ludGUyMTMxZmMyMzI4OHg=
     counter: "39145"
     sessionId: 041ce06d-43aa-491d-a160-40f7c2458a6f
   kind: ConfigMap
   metadata:
     annotations:
       control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"e49fde79-40c6-4f5e-93cf-df7051e75b50","leaseDuration":15.000000000,"acquireTime":"2020-11-01T07:44:39.833000Z","renewTime":"2020-11-02T05:29:44.871000Z","leaderTransitions":15661}'
     creationTimestamp: "2020-11-01T07:44:39Z"
     labels:
       app: k8s-ha-app-pr3
       configmap-type: high-availability
       type: flink-native-kubernetes
     name: k8s-ha-app-pr3-00000000000000000000000000000000-jobmanager-leader
     namespace: default
     resourceVersion: "1844956126"
     selfLink: /api/v1/namespaces/default/configmaps/k8s-ha-app-pr3-00000000000000000000000000000000-jobmanager-leader
     uid: 74163858-def8-473a-b7a3-34b66a5c6bd7
   ```
   
   
   ## Brief change log
   
   * Introduce CompletedCheckpointStore composition mechanism and migrate ZooKeeperCompletedCheckpointStore
   * Implement CheckpointRecoveryFactory based on Kubernetes API
   
   
   ## Verifying this change
   
   * All the changes are covered by new introduced unit tests.
   * I have also manually create a Flink cluster with K8s HA enabled in a real cluster.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (the doc will be added in a separate PR after the full feature merged)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518550184



##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;
+
+		private TestingRetrievableStateHandle(
+				T state,
+				FunctionWithException<T, T, IOException> retrieveStateFunction,
+				RunnableWithException discardStateRunnable,
+				Function<T, Long> getStateSizeFunction) {
 			this.state = state;
+			this.retrieveStateFunction = retrieveStateFunction;

Review comment:
       I am not fully understand here. Do you mean combining `retrieveStateFunction` and `getStateSizeFunction`? If it is, we have two question here.
   * How to get the size of retrieved `state`?
   * `getStateSize` do not throw any exception but `retrieveState` will.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518869980



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       I cannot explain why there is the `ConcurrentModificationException`. To me it looks that it is not really required.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 179472a87778a738c1f9f866156dcf6e024e69c7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202) 
   * 98a7d79ea52247e17fb1374f40de32109d56d63c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518533440



##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -30,9 +34,28 @@
  */
 public final class TestingRetrievableStateStorageHelper<T extends Serializable> implements RetrievableStateStorageHelper<T> {
 
+	private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+	private RunnableWithException discardStateRunnable;
+
+	private Function<T, Long> getStateSizeFunction;

Review comment:
       Yes, we may need to change these functions. For example, we want to throw an exception when `retrieveState`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   * dddbba5470b2766bc0703a2687e5b6200edb9fff UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518547824



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       1. IIUC, we will never throw a `ConcurrentModificationException` in `StateHandleStore#getAllAndLock`. So this retry does not take any effect.
   2. For `ZooKeeperStateHandleStore` implementation, it could retry internally to avoid concurrent modifications.
   3. For `KubernetesStateHandleStore` implementation, we do not need retry since we could get all the data in one time.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720247450






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   * dddbba5470b2766bc0703a2687e5b6200edb9fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163) 
   * 0d717493a2603c27fb06f1349f98d87de5473831 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-723518654


   @tillrohrmann @xintongsong Thanks all for the elaborative review and help with merging.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 270f2936428aac505152bf0e20129a4856ccb0ff Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762) 
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9239",
       "triggerID" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 98a7d79ea52247e17fb1374f40de32109d56d63c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9239) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9239",
       "triggerID" : "98a7d79ea52247e17fb1374f40de32109d56d63c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 179472a87778a738c1f9f866156dcf6e024e69c7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202) 
   * 98a7d79ea52247e17fb1374f40de32109d56d63c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9239) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   * dddbba5470b2766bc0703a2687e5b6200edb9fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-723388940


   Address the minor comments, rebase latest master and force pushed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518668943



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       After checking more previous commits, it seems that the retry loop in `ZooKeeperStateHandleStore#recover` is there from the first commit. Also in the `ZooKeeperStateHandleStore`, I do not notice that `getAllAndLock` or `getAllSortedByNameAndLock`(has replaced by the former) could throw a `ConcurrentModificationException` in all the history.
   
   @tillrohrmann Could you share some insights on this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann closed pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #13871:
URL: https://github.com/apache/flink/pull/13871


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518506150



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCheckpointStoreUtil.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link CheckpointStoreUtil} implementation for ZooKeeper.
+ *
+ */
+public class ZooKeeperCheckpointStoreUtil implements CheckpointStoreUtil {

Review comment:
       The class could be singleton.

##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointStoreUtil.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.runtime.checkpoint.CheckpointStoreUtil;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_ID_KEY_PREFIX;
+
+/**
+ * {@link CheckpointStoreUtil} implementation for Kubernetes.
+ *
+ */
+public class KubernetesCheckpointStoreUtil implements CheckpointStoreUtil {

Review comment:
       The class could be singleton.

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;
+
+		private TestingRetrievableStateHandle(
+				T state,
+				FunctionWithException<T, T, IOException> retrieveStateFunction,
+				RunnableWithException discardStateRunnable,
+				Function<T, Long> getStateSizeFunction) {
 			this.state = state;
+			this.retrieveStateFunction = retrieveStateFunction;

Review comment:
       I think we can combine these two arguments. The `state` can be covered by a `retrieveStateFunction` that always returns `state`.

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;

Review comment:
       I would vote for not allowing null values.
   We can provide no-op functions as default and do null checks in the setters in `TestingRetrievableStateStorageHelper`.

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;

Review comment:
       And +1 for making them `final` here.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       Trying to understand why the retry loop was needed before and not needed now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518116292



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -209,60 +191,39 @@ public void recover() throws Exception {
 		}
 	}
 
-	private boolean haveAllDownloaded(List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> checkpointPointers) {
-		if (completedCheckpoints.size() != checkpointPointers.size()) {
-			return false;
-		}
-		Set<Long> localIds = completedCheckpoints.stream().map(CompletedCheckpoint::getCheckpointID).collect(Collectors.toSet());
-		for (Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String> initialCheckpoint : checkpointPointers) {
-			if (!localIds.contains(pathToCheckpointId(initialCheckpoint.f1))) {
-				return false;
-			}
-		}
-		return true;
-	}
-
 	/**
-	 * Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
+	 * Synchronously writes the new checkpoints to state handle store and asynchronously removes older ones.
 	 *
 	 * @param checkpoint Completed checkpoint to add.
 	 */
 	@Override
-	public void addCheckpoint(final CompletedCheckpoint checkpoint, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception {
+	public void addCheckpoint(
+			final CompletedCheckpoint checkpoint,
+			CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception {

Review comment:
       nit: formatting of `postCleanup`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -209,60 +191,39 @@ public void recover() throws Exception {
 		}
 	}
 
-	private boolean haveAllDownloaded(List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> checkpointPointers) {
-		if (completedCheckpoints.size() != checkpointPointers.size()) {
-			return false;
-		}
-		Set<Long> localIds = completedCheckpoints.stream().map(CompletedCheckpoint::getCheckpointID).collect(Collectors.toSet());
-		for (Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String> initialCheckpoint : checkpointPointers) {
-			if (!localIds.contains(pathToCheckpointId(initialCheckpoint.f1))) {
-				return false;
-			}
-		}
-		return true;
-	}
-
 	/**
-	 * Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
+	 * Synchronously writes the new checkpoints to state handle store and asynchronously removes older ones.
 	 *
 	 * @param checkpoint Completed checkpoint to add.
 	 */
 	@Override
-	public void addCheckpoint(final CompletedCheckpoint checkpoint, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception {
+	public void addCheckpoint(
+			final CompletedCheckpoint checkpoint,
+			CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception {
+
 		checkNotNull(checkpoint, "Checkpoint");
 
-		final String path = checkpointIdToPath(checkpoint.getCheckpointID());
+		final String path = completedCheckpointStoreUtil.checkpointIDToName(checkpoint.getCheckpointID());
 
 		// Now add the new one. If it fails, we don't want to loose existing data.
-		checkpointsInZooKeeper.add(path, checkpoint);
+		checkpointStateHandleStore.add(path, checkpoint);
 
 		completedCheckpoints.addLast(checkpoint);
 
 		// Everything worked, let's remove a previous checkpoint if necessary.
 		while (completedCheckpoints.size() > maxNumberOfCheckpointsToRetain) {
 			final CompletedCheckpoint completedCheckpoint = completedCheckpoints.removeFirst();
-			tryRemoveCompletedCheckpoint(completedCheckpoint, completedCheckpoint.shouldBeDiscardedOnSubsume(), checkpointsCleaner, postCleanup);
+			tryRemoveCompletedCheckpoint(
+				completedCheckpoint,
+				completedCheckpoint.shouldBeDiscardedOnSubsume(),
+				checkpointsCleaner, postCleanup);

Review comment:
       nit: formatting of `postCleanup`.

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -30,9 +34,28 @@
  */
 public final class TestingRetrievableStateStorageHelper<T extends Serializable> implements RetrievableStateStorageHelper<T> {
 
+	private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+	private RunnableWithException discardStateRunnable;
+
+	private Function<T, Long> getStateSizeFunction;

Review comment:
       Can these fields be `final` or do we need to change them after having created a `TestingRetrievableStateStorageHelper` instance?

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;

Review comment:
       can these be `final`?

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStoreTest.java
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.testutils.FlinkMatchers;
+import org.apache.flink.runtime.persistence.StateHandleStore;
+import org.apache.flink.runtime.persistence.TestingStateHandleStore;
+import org.apache.flink.runtime.state.RetrievableStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.util.FlinkException;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
+
+/**
+ * Tests for {@link DefaultCompletedCheckpointStore}.
+ */
+public class DefaultCompletedCheckpointStoreTest {

Review comment:
       `extends TestLogger` is missing

##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;

Review comment:
       If the fields can be `null`, then we should add `@Nullable`.

##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointStoreUtil.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.runtime.checkpoint.CheckpointStoreUtil;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_ID_KEY_PREFIX;
+
+/**
+ * {@link CheckpointStoreUtil} implementation for Kubernetes.
+ *
+ */
+public class KubernetesCheckpointStoreUtil implements CheckpointStoreUtil {
+
+	private static final Logger LOG = LoggerFactory.getLogger(KubernetesCheckpointStoreUtil.class);
+
+	/**
+	 * Convert a checkpoint id into a ConfigMap key.
+	 *
+	 * @param checkpointId to convert to the key
+	 *
+	 * @return key created from the given checkpoint id
+	 */
+	@Override
+	public String checkpointIDToName(long checkpointId) {
+		return CHECKPOINT_ID_KEY_PREFIX + String.format("%019d", checkpointId);
+	}
+
+	/**
+	 * Converts a key in ConfigMap to the checkpoint id.
+	 *
+	 * @param key in ConfigMap
+	 *
+	 * @return Checkpoint id parsed from the key
+	 */
+	@Override
+	public long nameToCheckpointID(String key) {
+		try {
+			return Long.parseLong(key.replace(CHECKPOINT_ID_KEY_PREFIX, ""));

Review comment:
       `key.substring(CHECKPOINT_ID_KEY_PREFIX.length())` might be a bit faster.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCheckpointStoreUtil.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link CheckpointStoreUtil} implementation for ZooKeeper.
+ *
+ */
+public class ZooKeeperCheckpointStoreUtil implements CheckpointStoreUtil {
+
+	private static final Logger LOG = LoggerFactory.getLogger(ZooKeeperCheckpointStoreUtil.class);
+
+	/**
+	 * Convert a checkpoint id into a ZooKeeper path.
+	 *
+	 * @param checkpointId to convert to the path
+	 * @return Path created from the given checkpoint id
+	 */
+	@Override
+	public String checkpointIDToName(long checkpointId) {
+		return String.format("/%019d", checkpointId);
+	}
+
+	/**
+	 * Converts a path to the checkpoint id.
+	 *
+	 * @param path in ZooKeeper
+	 * @return Checkpoint id parsed from the path
+	 */
+	@Override
+	public long nameToCheckpointID(String path) {
+		try {
+			String numberString;
+
+			// check if we have a leading slash
+			if ('/' == path.charAt(0)) {
+				numberString = path.substring(1);
+			} else {
+				numberString = path;
+			}
+			return Long.parseLong(numberString);
+		} catch (NumberFormatException e) {
+			LOG.warn("Could not parse checkpoint id from {}. This indicates that the " +
+				"checkpoint id to path conversion has changed.", path);
+
+			return -1L;

Review comment:
       Maybe say in the JavaDocs of `CheckpointStoreUtil.nameToCheckpointID` that `-1L` represents an invalid checkpoint id or so.

##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointIDCounter.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMap;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesException;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector;
+import org.apache.flink.runtime.checkpoint.CheckpointIDCounter;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_COUNTER_KEY;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link CheckpointIDCounter} implementation for Kubernetes. The counter will be stored in
+ * JobManager-{@link org.apache.flink.api.common.JobID}-leader ConfigMap. The key is
+ * {@link org.apache.flink.kubernetes.utils.Constants#CHECKPOINT_COUNTER_KEY},
+ * and value is counter value.
+ */
+public class KubernetesCheckpointIDCounter implements CheckpointIDCounter {
+
+	private static final Logger LOG = LoggerFactory.getLogger(KubernetesCheckpointIDCounter.class);
+
+	private final FlinkKubeClient kubeClient;
+
+	private final String configMapName;
+
+	private final String lockIdentity;
+
+	private volatile boolean running;

Review comment:
       I think this class is not thread safe in the sense that a concurrent shutdown with a `getAndIncrement` call might not leave the config map with a cleaned up `CHECKPOINT_COUNTER_KEY` (e.g. if shutdown completes first and then `getAndIncrement` completes). Hence, I am wondering why `running` needs to be `volatile`?

##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -353,6 +360,40 @@ private static String getLogging(String logFile, String confDir, boolean hasLogb
 		return logging.toString();
 	}
 
+	/**
+	 * Create a {@link DefaultCompletedCheckpointStore} with {@link KubernetesStateHandleStore}.
+	 *
+	 * @param configuration configuration to build a RetrievableStateStorageHelper
+	 * @param kubeClient flink kubernetes client
+	 * @param configMapName ConfigMap name
+	 * @param executor executor to run blocking calls
+	 * @param lockIdentity lock identity to check the leadership
+	 * @param maxNumberOfCheckpointsToRetain max number of checkpoints to retain on state store handle
+	 *
+	 * @return a {@link DefaultCompletedCheckpointStore} with {@link KubernetesStateHandleStore}.
+	 *
+	 * @throws Exception when create the storage helper failed
+	 */
+	public static CompletedCheckpointStore createCompletedCheckpointStore(
+			Configuration configuration,
+			FlinkKubeClient kubeClient,
+			Executor executor,
+			String configMapName,
+			String lockIdentity,
+			int maxNumberOfCheckpointsToRetain) throws Exception {
+
+		final RetrievableStateStorageHelper<CompletedCheckpoint> stateStorage =
+			new FileSystemStateStorageHelper<>(HighAvailabilityServicesUtils
+				.getClusterHighAvailableStoragePath(configuration), COMPLETED_CHECKPOINT_FILE_SUFFIX);
+		final KubernetesStateHandleStore<CompletedCheckpoint> stateHandleStore = new KubernetesStateHandleStore<>(
+			kubeClient, configMapName, stateStorage, k -> k.startsWith(CHECKPOINT_ID_KEY_PREFIX), lockIdentity);
+		return new DefaultCompletedCheckpointStore(
+			maxNumberOfCheckpointsToRetain,
+			stateHandleStore,
+			new KubernetesCheckpointStoreUtil(),

Review comment:
       Could this be a singleton using `enum`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518580108



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       Just want to double check why we needed to catch `ConcurrentModificationException` at the first place, so we can be sure that removing the loop won't cause any problem.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518668943



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##########
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
 	}
 
 	/**
-	 * Gets the latest checkpoint from ZooKeeper and removes all others.
-	 *
-	 * <p><strong>Important</strong>: Even if there are more than one checkpoint in ZooKeeper,
-	 * this will only recover the latest and discard the others. Otherwise, there is no guarantee
-	 * that the history of checkpoints is consistent.
+	 * Recover all the valid checkpoints from state handle store. All the successfully recovered checkpoints will
+	 * be added to {@link #completedCheckpoints} sorted by checkpoint id.
 	 */
 	@Override
 	public void recover() throws Exception {
-		LOG.info("Recovering checkpoints from ZooKeeper.");
+		LOG.info("Recovering checkpoints from {}.", checkpointStateHandleStore);
 
 		// Get all there is first
-		List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints;
-		while (true) {
-			try {
-				initialCheckpoints = checkpointsInZooKeeper.getAll();
-				break;
-			}
-			catch (ConcurrentModificationException e) {
-				LOG.warn("Concurrent modification while reading from ZooKeeper. Retrying.");
-			}
-		}
+		final List<Tuple2<RetrievableStateHandle<CompletedCheckpoint>, String>> initialCheckpoints =
+			checkpointStateHandleStore.getAll();

Review comment:
       After checking more previous commits, it seems that the retry loop in `ZooKeeperStateHandleStore#recover` is there from the first commit. Also in the `ZooKeeperStateHandleStore`, I do not notice that `getAllAndLock` or `getAllSortedByNameAndLock`(has replaced by the former) could not throw a `ConcurrentModificationException` in all the history.
   
   @tillrohrmann Could you share some insights on this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dddbba5470b2766bc0703a2687e5b6200edb9fff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163) 
   * 0d717493a2603c27fb06f1349f98d87de5473831 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181) 
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518897446



##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointRecoveryFactory.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.runtime.checkpoint.CheckpointIDCounter;
+import org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory;
+import org.apache.flink.runtime.checkpoint.CompletedCheckpointStore;
+
+import java.util.concurrent.Executor;
+import java.util.function.Function;
+
+import static org.apache.flink.kubernetes.utils.KubernetesUtils.createCompletedCheckpointStore;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Factory to create {@link CompletedCheckpointStore} and {@link CheckpointIDCounter}.
+ */
+public class KubernetesCheckpointRecoveryFactory implements CheckpointRecoveryFactory {
+
+	private final FlinkKubeClient kubeClient;
+
+	private final Executor executor;
+
+	// Function to get the ConfigMap name for checkpoint. Input is job id, and output is ConfigMap name.
+	private final Function<JobID, String> getConfigMapNameFunction;
+
+	private final Configuration configuration;
+
+	private final String lockIdentity;
+
+	/**
+	 * Create a KubernetesCheckpointRecoveryFactory.
+	 *
+	 * @param kubeClient Kubernetes client
+	 * @param configuration Flink configuration
+	 * @param executor IO executor to run blocking calls
+	 * @param function Function to get the ConfigMap name for checkpoint.
+	 * @param lockIdentity Lock identity of current HA service
+	 */
+	public KubernetesCheckpointRecoveryFactory(
+			FlinkKubeClient kubeClient,
+			Configuration configuration,
+			Executor executor,
+			Function<JobID, String> function,
+			String lockIdentity) {
+
+		this.kubeClient = checkNotNull(kubeClient);
+		this.configuration = checkNotNull(configuration);
+		this.executor = checkNotNull(executor);
+		this.getConfigMapNameFunction = checkNotNull(function);
+		this.lockIdentity = checkNotNull(lockIdentity);
+	}
+
+	@Override
+	public CompletedCheckpointStore createCheckpointStore(
+			JobID jobID,
+			int maxNumberOfCheckpointsToRetain,
+			ClassLoader userClassLoader) throws Exception {
+
+		final String configMapName = getConfigMapNameFunction.apply(jobID);
+		return createCompletedCheckpointStore(

Review comment:
       nit: I am usually a bit against static method imports because it makes it harder to see where this method comes from. I think it is better to also put the class name of the method so that it is easily visible.

##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointIDCounter.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMap;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesException;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector;
+import org.apache.flink.runtime.checkpoint.CheckpointIDCounter;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_COUNTER_KEY;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link CheckpointIDCounter} implementation for Kubernetes. The counter will be stored in
+ * JobManager-{@link org.apache.flink.api.common.JobID}-leader ConfigMap. The key is
+ * {@link org.apache.flink.kubernetes.utils.Constants#CHECKPOINT_COUNTER_KEY},
+ * and value is counter value.
+ */
+public class KubernetesCheckpointIDCounter implements CheckpointIDCounter {
+
+	private static final Logger LOG = LoggerFactory.getLogger(KubernetesCheckpointIDCounter.class);
+
+	private final FlinkKubeClient kubeClient;
+
+	private final String configMapName;
+
+	private final String lockIdentity;
+
+	private boolean running;
+
+	public KubernetesCheckpointIDCounter(FlinkKubeClient kubeClient, String configMapName, String lockIdentity) {
+		this.kubeClient = checkNotNull(kubeClient);
+		this.configMapName = checkNotNull(configMapName);
+		this.lockIdentity = checkNotNull(lockIdentity);
+
+		this.running = false;
+	}
+
+	@Override
+	public void start() {
+		if (!running) {
+			running = true;
+		}
+	}
+
+	@Override
+	public void shutdown(JobStatus jobStatus) {
+		if (!running) {
+			return;
+		}
+		running = false;
+
+		LOG.info("Shutting down.");
+		if (jobStatus.isGloballyTerminalState()) {
+			LOG.info("Removing counter from ConfigMap {}", configMapName);
+			kubeClient.checkAndUpdateConfigMap(
+				configMapName,
+				configMap -> {
+					if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
+						configMap.getData().remove(CHECKPOINT_COUNTER_KEY);
+						return Optional.of(configMap);
+					}
+					return Optional.empty();
+				});
+		}
+	}
+
+	@Override
+	public long getAndIncrement() throws Exception {
+		final AtomicLong current = new AtomicLong();
+		final boolean updated = kubeClient.checkAndUpdateConfigMap(
+			configMapName,
+			configMap -> {
+				if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
+					final long currentValue = getCurrentCounter(configMap);
+					current.set(currentValue);
+					configMap.getData().put(CHECKPOINT_COUNTER_KEY, String.valueOf(currentValue + 1));
+					return Optional.of(configMap);
+				}
+				return Optional.empty();
+			}
+		).get();

Review comment:
       nit: line break could make it easier to read the code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d717493a2603c27fb06f1349f98d87de5473831 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181) 
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 179472a87778a738c1f9f866156dcf6e024e69c7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 179472a87778a738c1f9f866156dcf6e024e69c7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518538862



##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointIDCounter.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMap;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesException;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector;
+import org.apache.flink.runtime.checkpoint.CheckpointIDCounter;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_COUNTER_KEY;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link CheckpointIDCounter} implementation for Kubernetes. The counter will be stored in
+ * JobManager-{@link org.apache.flink.api.common.JobID}-leader ConfigMap. The key is
+ * {@link org.apache.flink.kubernetes.utils.Constants#CHECKPOINT_COUNTER_KEY},
+ * and value is counter value.
+ */
+public class KubernetesCheckpointIDCounter implements CheckpointIDCounter {
+
+	private static final Logger LOG = LoggerFactory.getLogger(KubernetesCheckpointIDCounter.class);
+
+	private final FlinkKubeClient kubeClient;
+
+	private final String configMapName;
+
+	private final String lockIdentity;
+
+	private volatile boolean running;

Review comment:
       Yes. You are right. Just like `DefaultCompletedCheckpointStore`, `KubernetesCheckpointIDCounter` is also not thread-safe. So `volatile` of `running` is unnecessary.
   
   BTW, I think `CheckpointCoordinator` could guarantee that there's no concurrent `shutdown` and `getAndIncrement`. I will simply remove the `volatile`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ab63312aba03160fffba96646c4f39257dc7f760",
       "triggerType" : "PUSH"
     }, {
       "hash" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202",
       "triggerID" : "179472a87778a738c1f9f866156dcf6e024e69c7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d717493a2603c27fb06f1349f98d87de5473831 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181) 
   * ab63312aba03160fffba96646c4f39257dc7f760 UNKNOWN
   * 179472a87778a738c1f9f866156dcf6e024e69c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9202) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518579393



##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##########
@@ -41,22 +64,43 @@
 
 		private final T state;
 
-		private TestingRetrievableStateHandle(T state) {
+		private FunctionWithException<T, T, IOException> retrieveStateFunction;
+
+		private RunnableWithException discardStateRunnable;
+
+		private Function<T, Long> getStateSizeFunction;
+
+		private TestingRetrievableStateHandle(
+				T state,
+				FunctionWithException<T, T, IOException> retrieveStateFunction,
+				RunnableWithException discardStateRunnable,
+				Function<T, Long> getStateSizeFunction) {
 			this.state = state;
+			this.retrieveStateFunction = retrieveStateFunction;

Review comment:
       No. I mean we do not need the argument `state` for the constructor.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wangyang0918 commented on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720248021


   cc @tillrohrmann @xintongsong @KarmaGYZ Please help with reviewing the last code changes PR at your convenience.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 270f2936428aac505152bf0e20129a4856ccb0ff Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762) 
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 962ecfd11019d4d6a40a7dd062028847cec3d223 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096) 
   * dddbba5470b2766bc0703a2687e5b6200edb9fff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163) 
   * 0d717493a2603c27fb06f1349f98d87de5473831 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13871:
URL: https://github.com/apache/flink/pull/13871#issuecomment-720254759


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8750",
       "triggerID" : "4404df5031c62c3c57d948766dd4a996e4773372",
       "triggerType" : "PUSH"
     }, {
       "hash" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8762",
       "triggerID" : "270f2936428aac505152bf0e20129a4856ccb0ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9096",
       "triggerID" : "962ecfd11019d4d6a40a7dd062028847cec3d223",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163",
       "triggerID" : "dddbba5470b2766bc0703a2687e5b6200edb9fff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181",
       "triggerID" : "0d717493a2603c27fb06f1349f98d87de5473831",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dddbba5470b2766bc0703a2687e5b6200edb9fff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9163) 
   * 0d717493a2603c27fb06f1349f98d87de5473831 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9181) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org