You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/10 03:49:42 UTC
[GitHub] [flink] wangyang0918 opened a new pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
wangyang0918 opened a new pull request #14006:
URL: https://github.com/apache/flink/pull/14006
Add documentation for FLIP-144(native Kubernetes HA).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
}, {
"hash" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1793e46f94174a314f182dbf7845cd229d014d70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462)
* 19fdb8c1ce84f1a0c8b432331e8da5678fd98383 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726174856
One question, can you use k8s ha with a standalone deployment on top of k8s or only with the native k8s resource manager?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r523933632
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
> I don't understand the subclause , as to not occupy .... resources.
What I want to state here is that we will free all the allocated resources from K8s so that they could be used by others. At the same time, we still have the ability to recover the Flink application at any time if you want.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726237834
I think it should also work with a standalone deployment. But I haven't tried it out myself. Could be a good testing task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #14006:
URL: https://github.com/apache/flink/pull/14006
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r523061199
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
maybe: deploy -> deployment in the text?
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
maybe: will be deleted
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
For future documentation work: It is simpler to review if you put every sentence in its own line. That way it is much easier to see where the comment belongs to.
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
HA related ConfigMaps will be retained because they do not set the owner reference.
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
When restarting the session / application using `kubernetes-session.sh` or `flink run-application`, all previously running jobs will be recovered and restart from the latest successful checkpoint.
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
I don't understand the subclause `, as to not occupy .... resources`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724432808
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1ee231fa8559cdc63ec4956262da92496698826a (Tue Nov 10 03:52:18 UTC 2020)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ee231fa8559cdc63ec4956262da92496698826a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726475387
@sjwiesman Thanks for raising up this question.
Before writing the documentation, I have verified that the Kubernetes HA service could work both for standalone and native
K8s cluster. And that is by design.
Also I have added two examples in the documentation about how to start a standalone or native cluster with HA configured. Do you think it is not clear enough?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-727789055
I have created a new ticket FLINK-20167 for the follow-up(rework the HA documentation).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r524077381
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
Yeah. I have addressed the comments and force pushed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r523933632
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
> I don't understand the subclause , as to not occupy .... resources.
What I want to state here is that we will free all the allocated resources from K8s so that they could be used by others. At the same time, we still have the ability to recover the Flink application at any time if you want. Do you have some suggestions to make it more clear?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
}, {
"hash" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487",
"triggerID" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19fdb8c1ce84f1a0c8b432331e8da5678fd98383 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r523061517
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
maybe: will be destroyed -> will be deleted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
}, {
"hash" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487",
"triggerID" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"triggerType" : "PUSH"
}, {
"hash" : "d8a9091c684268dac87a5410755b4a49aa951017",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8a9091c684268dac87a5410755b4a49aa951017",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19fdb8c1ce84f1a0c8b432331e8da5678fd98383 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487)
* d8a9091c684268dac87a5410755b4a49aa951017 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
}, {
"hash" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487",
"triggerID" : "19fdb8c1ce84f1a0c8b432331e8da5678fd98383",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1793e46f94174a314f182dbf7845cd229d014d70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462)
* 19fdb8c1ce84f1a0c8b432331e8da5678fd98383 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9487)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-725341672
Address the comments and force pushed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r523063603
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
When restarting the session / application using `kubernetes-session.sh` or `flink run-application`, all previously running jobs will be recovered and restarted from the latest successful checkpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ee231fa8559cdc63ec4956262da92496698826a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375)
* 1793e46f94174a314f182dbf7845cd229d014d70 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726250727
Hmm, ok @tillrohrmann and @wangyang0918 I will defer to the two of you. I read the docs right now as being ambiguous as to whether or not k8s ha works with standalone clusters. I think what's here is fine to merge as is but maybe make a note to come back and update the docs if we find it doesn't work for standalone during release testing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r521519370
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,74 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
Review comment:
This shouldn't be Hadoop specific.
```suggestion
<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured ha storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
```
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,74 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
Review comment:
I'm a little confused by this sentence, who is "we" referring to? Does this make sense?
```suggestion
When running Flink as a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), the `replica` count should be configured to 1.
This means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
```
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,74 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+If the user wants to keep the HA data and restart the Flink cluster, he/she could also simply delete the deploy(via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed(e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap) so that it will not occupy the Kubernetes cluster resources. For the HA related ConfigMaps, we do not set the owner reference and they could be retained. Then he/she could use `kubernetes-session.sh` or `flink run-application` to start the session/application again. All the previous suspending running jobs could recover from the latest checkpoint successfully.
Review comment:
A good rule of thumb, if you ever find yourself writing "he/she" then just use "they" instead.
Also, the word "could" implies that something only works some of the time but I don't think that's what you mean to say here.
```suggestion
To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
```
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,74 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
Review comment:
```suggestion
Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ee231fa8559cdc63ec4956262da92496698826a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ee231fa8559cdc63ec4956262da92496698826a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r524047117
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
I think we don't need the `as to not occupy ... resources` subclause. Are you also addressing my other comments for this paragraph?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r521791380
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,74 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
Review comment:
Hmm, I think "running Flink as a Kubernetes deployment" is not accurate. Because we are running Flink JobManager as a Kubernetes deployment. Do you think the following sentences make sense?
When running Flink JobManager as a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), the `replica` count should be configured to 1 or greater.
* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
* The value `N`(greater than 1) means that multiple JobManagers will be launched while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r524048796
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,76 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+When running Flink JobManager as a Kubernetes deployment, the replica count should be configured to 1 or greater.
+* The value `1` means that a new JobManager will be launched to take over leadership if the current one terminates exceptionally.
+* The value `N` (greater than 1) means that multiple JobManagers will be launched simultaneously while one is active and others are standby. Starting more than one JobManager will make the recovery faster.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High Availability Data Clean Up
+Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA state on DFS, will be cleaned up.
+
+So the following command will only shut down the Flink session cluster and leave all the HA related ConfigMaps, state untouched.
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+The following commands will cancel the job in application or session cluster and effectively remove all its HA data.
+{% highlight bash %}
+# Cancel a Flink job in the existing session
+$ ./bin/flink cancel -t kubernetes-session -Dkubernetes.cluster-id=<ClusterID> <JobID>
+# Cancel a Flink application
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
+{% endhighlight %}
+
+To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be destroyed (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap), as to not occupy the Kubernetes cluster resources. However, HA related ConfigMaps do not set the owner reference and they will be retained. When restarting the session / application, use `kubernetes-session.sh` or `flink run-application`. All the previous suspending running jobs will recover from the latest checkpoint successfully.
Review comment:
```suggestion
To keep HA data while restarting the Flink cluster, simply delete the deploy (via `kubectl delete deploy <ClusterID>`). All the Flink cluster related resources will be deleted (e.g. JobManager Deployment, TaskManager pods, services, Flink conf ConfigMap). HA related ConfigMaps will be retained because they do not set the owner reference. When restarting the session / application using `kubernetes-session.sh` or `flink run-application`, all previously running jobs will be recovered and restarted from the latest successful checkpoint.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ee231fa8559cdc63ec4956262da92496698826a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375)
* 1793e46f94174a314f182dbf7845cd229d014d70 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724434807
cc @tillrohrmann @sjwiesman Could you please have a look on the documentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r521212558
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,69 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High availability data clean up
+Currently, when a Flink cluster reached the terminal state(`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA storage on DFS, will be cleaned up. So if the user stops the Flink session/application cluster manually(via the following two commands), the Flink cluster could not be recovered.
+
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
Review comment:
Hmm. After checking the implementation, the above command will only shut down the cluster and not clean up the HA state. And we are not deleting the HA related ConfigMap here.
Only when the job reached to the terminal state, the HA data will be cleaned up.
I will update the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726478193
@wangyang0918 could you please just add a sentence that explicitly states it works for both standalone and native resource manager. It wasn’t obvious to me. After that +1 to merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726874200
@tillrohrmann sounds good. We can discuss that as a follow up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724434807
cc @tillrohrmann Could you please have a look on the documentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-724437611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ee231fa8559cdc63ec4956262da92496698826a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9375",
"triggerID" : "1ee231fa8559cdc63ec4956262da92496698826a",
"triggerType" : "PUSH"
}, {
"hash" : "1793e46f94174a314f182dbf7845cd229d014d70",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462",
"triggerID" : "1793e46f94174a314f182dbf7845cd229d014d70",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1793e46f94174a314f182dbf7845cd229d014d70 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9462)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14006:
URL: https://github.com/apache/flink/pull/14006#discussion_r520462834
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,69 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High availability data clean up
+Currently, when a Flink cluster reached the terminal state(`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA storage on DFS, will be cleaned up. So if the user stops the Flink session/application cluster manually(via the following two commands), the Flink cluster could not be recovered.
Review comment:
```suggestion
Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA storage on DFS, will be cleaned up. So if the user stops the Flink session/application cluster manually(via the following two commands), the Flink cluster could not be recovered.
```
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,69 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High availability data clean up
+Currently, when a Flink cluster reached the terminal state(`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA storage on DFS, will be cleaned up. So if the user stops the Flink session/application cluster manually(via the following two commands), the Flink cluster could not be recovered.
+
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
+{% endhighlight %}
+
+{% highlight bash %}
+$ ./bin/flink cancel -t kubernetes-application -Dkubernetes.cluster-id=<ClusterID> <JobID>
Review comment:
This will cancel the job and effectively remove all its HA data.
##########
File path: docs/ops/jobmanager_high_availability.md
##########
@@ -215,6 +215,69 @@ Starting zookeeper daemon on host localhost.</pre>
<pre>
$ bin/yarn-session.sh -n 2</pre>
+## Kubernetes Cluster High Availability
+We are using a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the JobManager when deploying Flink on Kubernetes cluster. The `replicas` is configured to 1, which means that a new JobManager will be launched to take over the leadership once the current one terminated exceptionally.
+
+### Configuration
+{% highlight yaml %}
+kubernetes.cluster-id: <ClusterId>
+high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+high-availability.storageDir: hdfs:///flink/recovery
+{% endhighlight %}
+
+#### Example: Highly Available Standalone Flink Cluster on Kubernetes
+Both session and job/application clusters could use the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml]({{ site.baseurl}}/ops/deployment/kubernetes.html#common-cluster-resource-definitions). All other yamls do not need to be updated.
+
+<span class="label label-info">Note</span> You need to have hadoop in the classpath or enable the corresponding FileSystem plugin for the Flink image. Refer to [custom Flink image]({{ site.baseurl}}/ops/deployment/docker.html#customize-flink-image) and [enable plugins]({{ site.baseurl}}/ops/deployment/docker.html#using-plugins) for more information.
+
+{% highlight yaml %}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: flink-config
+ labels:
+ app: flink
+data:
+ flink-conf.yaml: |+
+ ...
+ kubernetes.cluster-id: <ClusterId>
+ high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
+ high-availability.storageDir: hdfs:///flink/recovery
+ restart-strategy: fixed-delay
+ restart-strategy.fixed-delay.attempts: 10
+ ...
+{% endhighlight %}
+
+#### Example: Highly Available Native Kubernetes Cluster
+Using the following command to start a native Flink application cluster on Kubernetes with high availability configured.
+{% highlight bash %}
+$ ./bin/flink run-application -p 8 -t kubernetes-application \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dkubernetes.container.image=<CustomImageName> \
+ -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
+ -Dhigh-availability.storageDir=s3://flink/flink-ha \
+ -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
+ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar \
+ local:///opt/flink/examples/streaming/StateMachineExample.jar
+{% endhighlight %}
+
+### High availability data clean up
+Currently, when a Flink cluster reached the terminal state(`FAILED`, `CANCELED`, `FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA storage on DFS, will be cleaned up. So if the user stops the Flink session/application cluster manually(via the following two commands), the Flink cluster could not be recovered.
+
+{% highlight bash %}
+$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true
Review comment:
Shouldn't this simply shut down the cluster which suspends all running jobs? Shouldn't this leave the HA state untouched? Or is it because we are deleting all the HA config maps here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14006: [FLINK-19546][doc] Add documentation for native Kubernetes HA
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14006:
URL: https://github.com/apache/flink/pull/14006#issuecomment-726250727
Hmm, ok @tillrohrmann and @wangyang0918 I will defer to the two of you. I read the docs right now as being ambiguous as to whether or not k8s ha works with standalone clusters. I think what's here is fine to merge as is but maybe make a note to come back and update the docs if we find it doesn't work during release testing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org