You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/30 08:00:24 UTC
[GitHub] [flink-kubernetes-operator] wangyang0918 opened a new pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
wangyang0918 opened a new pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133
After shutting down the cluster with HA ConfigMap retained, we could simply delete the JobGraph information in the ConfigMap. When the FlinkDeployment is resumed, the EmbeddedExecutor will generate a new JobGraph, which picks up the latest job config changes, and then submit the job. It could also recover from the latest checkpoint.
This is a short-term but not temporary hack and it could work both for 1.14 and 1.15.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
gyfora commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838294502
##########
File path: e2e-tests/test_last_state_upgrade.sh
##########
@@ -44,6 +44,17 @@ function wait_for_jobmanager_running() {
wait_for_logs $jm_pod_name "Rest endpoint listening at" ${TIMEOUT} || exit 1
}
+function assert_available_slots() {
+ expected=$1
+ ip=$(minikube ip)
+ actual=$(curl http://$ip/default/${CLUSTER_ID}/overview 2>/dev/null | grep -E -o '"slots-available":[0-9]+' | awk -F':' '{print $2}')
Review comment:
wouldnt slots available depend on TM config only? that used to work even before I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
Posted by GitBox <gi...@apache.org>.
gyfora commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838243123
##########
File path: flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java
##########
@@ -202,14 +206,18 @@ public boolean isJobManagerPortReady(Configuration config) {
}
break;
case LAST_STATE:
- final String namespace = conf.getString(KubernetesConfigOptions.NAMESPACE);
FlinkUtils.deleteCluster(namespace, clusterId, kubernetesClient, false);
break;
default:
throw new RuntimeException("Unsupported upgrade mode " + upgradeMode);
}
}
- FlinkUtils.waitForClusterShutdown(kubernetesClient, conf);
+ FlinkUtils.waitForClusterShutdown(kubernetesClient, namespace, clusterId);
+ if (HighAvailabilityMode.isHighAvailabilityModeActivated(conf)) {
+ // Delete the job graph in the HA ConfigMaps so that the newly changed job config(e.g.
+ // parallelism) could take effect
+ FlinkUtils.deleteJobGraphInKubernetesHA(clusterId, namespace, kubernetesClient);
+ }
Review comment:
Wouldn't it be more safe to execute this logic before deployment? Technically we could have an error after shutting down the job but before deleting the jobgraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#issuecomment-1082760661
cc @gyfora
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
gyfora commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838299302
##########
File path: e2e-tests/test_last_state_upgrade.sh
##########
@@ -44,6 +44,17 @@ function wait_for_jobmanager_running() {
wait_for_logs $jm_pod_name "Rest endpoint listening at" ${TIMEOUT} || exit 1
}
+function assert_available_slots() {
+ expected=$1
+ ip=$(minikube ip)
+ actual=$(curl http://$ip/default/${CLUSTER_ID}/overview 2>/dev/null | grep -E -o '"slots-available":[0-9]+' | awk -F':' '{print $2}')
Review comment:
ah okay, I misunderstood the check. good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838253938
##########
File path: flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java
##########
@@ -202,14 +206,18 @@ public boolean isJobManagerPortReady(Configuration config) {
}
break;
case LAST_STATE:
- final String namespace = conf.getString(KubernetesConfigOptions.NAMESPACE);
FlinkUtils.deleteCluster(namespace, clusterId, kubernetesClient, false);
break;
default:
throw new RuntimeException("Unsupported upgrade mode " + upgradeMode);
}
}
- FlinkUtils.waitForClusterShutdown(kubernetesClient, conf);
+ FlinkUtils.waitForClusterShutdown(kubernetesClient, namespace, clusterId);
+ if (HighAvailabilityMode.isHighAvailabilityModeActivated(conf)) {
+ // Delete the job graph in the HA ConfigMaps so that the newly changed job config(e.g.
+ // parallelism) could take effect
+ FlinkUtils.deleteJobGraphInKubernetesHA(clusterId, namespace, kubernetesClient);
+ }
Review comment:
I think your suggestion makes sense. I will update the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
gyfora commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838302558
##########
File path: e2e-tests/test_last_state_upgrade.sh
##########
@@ -44,6 +44,17 @@ function wait_for_jobmanager_running() {
wait_for_logs $jm_pod_name "Rest endpoint listening at" ${TIMEOUT} || exit 1
}
+function assert_available_slots() {
+ expected=$1
+ ip=$(minikube ip)
+ actual=$(curl http://$ip/default/${CLUSTER_ID}/overview 2>/dev/null | grep -E -o '"slots-available":[0-9]+' | awk -F':' '{print $2}')
Review comment:
yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838298288
##########
File path: e2e-tests/test_last_state_upgrade.sh
##########
@@ -44,6 +44,17 @@ function wait_for_jobmanager_running() {
wait_for_logs $jm_pod_name "Rest endpoint listening at" ${TIMEOUT} || exit 1
}
+function assert_available_slots() {
+ expected=$1
+ ip=$(minikube ip)
+ actual=$(curl http://$ip/default/${CLUSTER_ID}/overview 2>/dev/null | grep -E -o '"slots-available":[0-9]+' | awk -F':' '{print $2}')
Review comment:
I do not think so. If the parallelism is configured 2, we will get 0 available slot after the job is running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#issuecomment-1082778223
Pushed a `fixup` to address the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on a change in pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#discussion_r838301360
##########
File path: e2e-tests/test_last_state_upgrade.sh
##########
@@ -44,6 +44,17 @@ function wait_for_jobmanager_running() {
wait_for_logs $jm_pod_name "Rest endpoint listening at" ${TIMEOUT} || exit 1
}
+function assert_available_slots() {
+ expected=$1
+ ip=$(minikube ip)
+ actual=$(curl http://$ip/default/${CLUSTER_ID}/overview 2>/dev/null | grep -E -o '"slots-available":[0-9]+' | awk -F':' '{print $2}')
Review comment:
I think you mean the "slots-total" :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 edited a comment on pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when shutting down cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133#issuecomment-1082778223
Force pushed to address the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 merged pull request #133: [FLINK-26916] Remove job graph from K8s HA ConfigMaps when submitting a Flink application cluster
Posted by GitBox <gi...@apache.org>.
wangyang0918 merged pull request #133:
URL: https://github.com/apache/flink-kubernetes-operator/pull/133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org