You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/01 17:48:51 UTC
[GitHub] [pulsar] michaeljmarshall opened a new issue #8770: SIGTERM Not Forwarded to Component Process for `bin/pulsar` Script, Preventing Clean Shutdown
michaeljmarshall opened a new issue #8770:
URL: https://github.com/apache/pulsar/issues/8770
I am deploying a pulsar cluster on Kubernetes using the `apachepulsar/pulsar-all:2.6.2` docker image. I start each of the component containers using the `bin/pulsar` script. This is the same method used by the pulsar helm chart.
I've noticed that when stopping any of the component pods (Zookeeper, Broker, Bookkeeper), they do not run their shutdown hooks, but are instead killed forcefully after the configured grace period. In looking at the `bin/pulsar` script, I can see that the script ends by running `exec` on the java process without trapping the SIGTERM sent to the container and forwarding it to the process's pid. For example, here is the script that starts a bookie pod:
https://github.com/apache/pulsar/blob/411bf71006039af661be13c54dda4ce074f97679/bin/pulsar#L312
Because we don't forward it, the java process never receives the SIGTERM notification. I'd like to propose modifying the `bin/pulsar` script to properly trap the SIGTERM and pass it to the process.
I'm happy to submit a PR if there is support for this idea.
Additionally, I noticed that there is a `bin/pulsar-daemon` script that starts a component process and writes the pid to a file in an attempt to properly manage the communicate the SIGTERM. However, the way that script is written, it exits after dispatching the component process, which leads the container to exit. It seems like this script would have a different use case than running in a container. Further, I don't see a use case where someone would want to run any of the components using `bin/pulsar` without being able to cleanly shut them down.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #8770:
URL: https://github.com/apache/pulsar/issues/8770#issuecomment-736741873
@michaeljmarshall It might be an issue in the deployment too. I recently fixed a related problem in the helm charts with https://github.com/apache/pulsar-helm-chart/pull/59 . Please check that your deployment doesn't block the signals.
What is the problem in using `exec`? I'd assume that it's the best practice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on issue #8770:
URL: https://github.com/apache/pulsar/issues/8770#issuecomment-736741873
@michaeljmarshall It might be an issue in the deployment too. I recently fixed a related problem in the helm charts with https://github.com/apache/pulsar-helm-chart/pull/59 . Please check that your Pulsar helm chart contains those changes.
What is the problem in using `exec`? I'd assume that it's the best practice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #8770:
URL: https://github.com/apache/pulsar/issues/8770#issuecomment-736749343
@lhotari - thanks for your quick comment. You're right that the issue was with my deployment, and that by changing `bin/pulsar bookie` to `exec bin/pulsar bookie`, the sigterm is properly handled by the component process.
In case anyone else runs across this, I started with this:
```yaml
spec:
containers:
- args:
- bin/apply-config-from-env.py conf/broker.conf && bin/apply-config-from-env.py
conf/pulsar_env.sh && bin/gen-yml-from-env.py conf/functions_worker.yml
&& bin/pulsar broker
command:
- sh
- -c
```
and changed it to this:
```yaml
spec:
containers:
- args:
- bin/apply-config-from-env.py conf/broker.conf && bin/apply-config-from-env.py
conf/pulsar_env.sh && bin/gen-yml-from-env.py conf/functions_worker.yml
&& exec bin/pulsar broker
command:
- sh
- -c
```
and now it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #8770:
URL: https://github.com/apache/pulsar/issues/8770#issuecomment-736715562
The same improvement could be made to the [bin/bookkeeper](https://github.com/apache/bookkeeper/blob/master/bin/bookkeeper) script in the bookkeeper project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on issue #8770:
URL: https://github.com/apache/pulsar/issues/8770#issuecomment-736741873
@michaeljmarshall It might be an issue in the deployment too. I recently fixed a related problem in the helm charts with https://github.com/apache/pulsar-helm-chart/pull/59 . Please check that your Pulsar helm chart contains [those changes.](https://github.com/apache/pulsar-helm-chart/commit/6c2edba8b1c3e56b8b2bdc2689004fe88637db28)
What is the problem in using `exec`? I'd assume that it's the best practice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall closed issue #8770: SIGTERM Not Forwarded to Component Process for bin/pulsar Script, Preventing Clean Shutdown
Posted by GitBox <gi...@apache.org>.
michaeljmarshall closed issue #8770:
URL: https://github.com/apache/pulsar/issues/8770
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org