You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "SchneiderV (via GitHub)" <gi...@apache.org> on 2023/08/07 06:31:41 UTC
[GitHub] [pulsar-helm-chart] SchneiderV opened a new issue, #373: wait-zookeeper-ready init container OOM killed
SchneiderV opened a new issue, #373:
URL: https://github.com/apache/pulsar-helm-chart/issues/373
**Describe the bug**
The broker and proxy pods does not start, because their wait-zookeeper-ready init container OOM kills the zookeeper client.
**To Reproduce**
Steps to reproduce the behavior:
1. A working Kubernetes Cluster with a pulsar 2.10.x
2. Configure a few topics and subscriptions
3. Send messages through pulsar
4. Then upgrade to 2.11.x or 3.0.0
5. Everything starts as expected, beside the broker and proxies
6. The log of the broker/proxy states repeatedly
```
pulsar cluster <cluster-name> isn't initialized yet ... check in 3 seconds ...
Killed
```
However, the zookeepers are up and running
**Expected behavior**
The zookeeper client in the wait-zookeeper-ready init container is not OOM killed but works correctly
**Additional context**
The problem is, that for the zookeeper client, the args `-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g` are set.
However, the init container does only have 512mb memory.
In detail:
- The wait-zookeeper-ready init container calls `bin/pulsar zookeeper-shell -server <cluster-name>-zookeeper get /admin/clusters/<cluster-name>` (proxy) or `bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server <cluster-name>-zookeeper:2181 get /admin/clusters/<cluster-name>` (broker)
- The bin/pulsar or bin/bookkeeper script calls bkenv.sh (https://github.com/apache/pulsar/blob/3eadbc3a437b84a0a04fc926c268c4003feda81e/conf/bkenv.sh#L38) to prepare the JVM arguments
- The bkenv.sh (line 38) sets the arguments for 2g of memory: `BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g"}}`
- At startup, the JVM of the zookeeper client tries to allocate around 1.7g of memory and is OOM Killed by the init container operating system.
**Workaround**
Use a helm post-renderer and add `export BOOKIE_MEM="-Xms256m -Xmx512m -XX:MaxDirectMemorySize=512m";` to the args of the wait-zookeeper-ready container
**Intention**
Please fix the helm-chart in some way, that the zookeeper client is not called with the args `-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tisonkun closed issue #373: wait-zookeeper-ready init container OOM killed
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun closed issue #373: wait-zookeeper-ready init container OOM killed
URL: https://github.com/apache/pulsar-helm-chart/issues/373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tisonkun commented on issue #373: wait-zookeeper-ready init container OOM killed
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #373:
URL: https://github.com/apache/pulsar-helm-chart/issues/373#issuecomment-1667264708
Sorry. I misread the repo. This is exactly where we should handle it. You can submit a PR and ping me for review - I may not have enough time recently to work out a solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] tisonkun commented on issue #373: wait-zookeeper-ready init container OOM killed
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #373:
URL: https://github.com/apache/pulsar-helm-chart/issues/373#issuecomment-1667263934
Thanks for your report!
This seems an issue of the Helm Chart and we have https://github.com/apache/pulsar-helm-chart/issues/124 for it. Let's discuss there - nothing can be done in the main repo for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] wait-zookeeper-ready init container OOM killed [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed issue #373: wait-zookeeper-ready init container OOM killed
URL: https://github.com/apache/pulsar-helm-chart/issues/373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org