You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/10/26 22:16:20 UTC
[GitHub] [pulsar] dlg99 opened a new pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
dlg99 opened a new pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504
### Motivation
Restart of connector via `pulsar-admin source restart` (debezium postgres, but reprod with another too) failed and the function worker became non-responsive, repeatedly logging
```
19:43:58.139 [function-web-27-6] ERROR org.apache.pulsar.functions.worker.rest.api.ComponentImpl - Failed to restart Source: public/default/cassandra-source-ks1-table1
org.apache.pulsar.client.admin.PulsarAdminException$TimeoutException: java.util.concurrent.TimeoutException
at org.apache.pulsar.client.admin.internal.SourcesImpl.restartSource(SourcesImpl.java:474) ~[com.datastax.oss-pulsar-client-admin-original-2.7.2.1.1.8-SNAPSHOT.jar:2.7.2.1.1.8-SNAPSHOT]
at org.apache.pulsar.functions.worker.FunctionRuntimeManager.restartFunctionInstances(FunctionRuntimeManager.java:427) ~[com.datastax.oss-pulsar-functions-worker-2.7.2.1.1.8-SNAPSHOT.jar:2.7.2.1.1.8-SNAPSHOT]
at org.apache.pulsar.functions.worker.rest.api.ComponentImpl.restartFunctionInstances(ComponentImpl.java:699) [com.datastax.oss-pulsar-functions-worker-2.7.2.1.1.8-SNAPSHOT.jar:2.7.2.1.1.8-SNAPSHOT]
at org.apache.pulsar.functions.worker.rest.api.v3.SourcesApiV3Resource.restartSource(SourcesApiV3Resource.java:187) [com.datastax.oss-pulsar-functions-worker-2.7.2.1.1.8-SNAPSHOT.jar:2.7.2.1.1.8-SNAPSHOT]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
...
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.client.admin.internal.SourcesImpl.restartSource(SourcesImpl.java:467) ~[com.datastax.oss-pulsar-client-admin-original-2.7.2.1.1.8-SNAPSHOT.jar:2.7.2.1.1.8-SNAPSHOT]
```
### Modifications
the rootcause tracked to the k8s client call timing out.
Looks like V1DeleteOptions weren't passed to the corresponding calls, and `Foreground` policy was not passed properly AFAICT from the k8s-client github/issues.
I also set grace period to zero for deleteNamespacedStatefulSetCall (similar to deleteNamespacedServiceCall)
### Verifying this change
Tested on the env.
Don't know how to unit test this.
### Does this pull request potentially affect one of the following parts:
NO, AFAIK.
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
- [X] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli merged pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
eolivelli merged pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504#discussion_r737056965
##########
File path: pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
##########
@@ -567,7 +567,12 @@ private void submitStatefulSet() throws Exception {
public void deleteStatefulSet() throws InterruptedException {
String statefulSetName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);
final V1DeleteOptions options = new V1DeleteOptions();
- options.setGracePeriodSeconds(5L);
+ // gracePeriodSeconds == 0 to force pod deletion and
+ // avoid cases when pod hangs during termination
+ // https://amalgjose.com/2021/07/28/how-to-delete-a-kubernetes-pod-which-is-stuck-in-terminating-state/
+ // https://www.ibm.com/support/pages/kubernetes-pods-are-stuck-terminating-state
+ // https://github.com/kubernetes-client/java/issues/770
+ options.setGracePeriodSeconds(0L);
Review comment:
Can we make the grace period configurable as part of the k8s runtime factory config:
https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactoryConfig.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504#discussion_r738000371
##########
File path: site2/docs/functions-runtime.md
##########
@@ -136,6 +136,10 @@ functionRuntimeFactoryConfigs:
extraFunctionDependenciesDir:
# Additional memory padding added on top of the memory requested by the function per on a per instance basis
percentMemoryPadding: 10
+ # The duration in seconds before the StatefulSet deleted on function stop/restart.
+ # Value must be non-negative integer. The value zero indicates delete immediately.
Review comment:
```suggestion
# Value must be a non-negative integer. 0 indicates the StatefulSet is deleted immediately.
```
do you mean this?
##########
File path: site2/docs/functions-runtime.md
##########
@@ -136,6 +136,10 @@ functionRuntimeFactoryConfigs:
extraFunctionDependenciesDir:
# Additional memory padding added on top of the memory requested by the function per on a per instance basis
percentMemoryPadding: 10
+ # The duration in seconds before the StatefulSet deleted on function stop/restart.
Review comment:
```suggestion
# The duration (in seconds) before the StatefulSet is deleted after a function stops or restarts.
```
do you mean this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504#issuecomment-954065989
@jerrypeng @Anonymitaet Please take another look; I addressed your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504#discussion_r737100648
##########
File path: pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
##########
@@ -567,7 +567,12 @@ private void submitStatefulSet() throws Exception {
public void deleteStatefulSet() throws InterruptedException {
String statefulSetName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);
final V1DeleteOptions options = new V1DeleteOptions();
- options.setGracePeriodSeconds(5L);
+ // gracePeriodSeconds == 0 to force pod deletion and
+ // avoid cases when pod hangs during termination
+ // https://amalgjose.com/2021/07/28/how-to-delete-a-kubernetes-pod-which-is-stuck-in-terminating-state/
+ // https://www.ibm.com/support/pages/kubernetes-pods-are-stuck-terminating-state
+ // https://github.com/kubernetes-client/java/issues/770
+ options.setGracePeriodSeconds(0L);
Review comment:
@jerrypeng fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #12504: k8s runtime: force deletion to avoid hung function worker during connector restart
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #12504:
URL: https://github.com/apache/pulsar/pull/12504#issuecomment-954363711
@dlg99 LGTM from a tech writing perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org