You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/17 12:32:00 UTC
[GitHub] [pulsar] Maxzouz opened a new issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Maxzouz opened a new issue #11687:
URL: https://github.com/apache/pulsar/issues/11687
#### Expected behavior
RAM usage of pulsar bookie pods shouldn't exceeds xms/xmx values
#### Actual behavior
RAM usage is way over the limits fixed in config file
#### Steps to reproduce
S/O
#### System configuration
**Pulsar version**: 2.6.1
**K8s version**: v1.17.5
Hello,
I'm working on a pulsar cluster integrated inside a Rancher kubernetes cluster and deployed via helm.
Here's the values of the bookie configmap :
> PULSAR_GC: |
-XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -Xloggc:/var/log/bookie-gc.log -XX:G1LogLevel=finest
PULSAR_MEM: |
-Xms3g -Xmx4g -XX:MaxDirectMemorySize=2g
PULSAR_PREFIX_journalDirectories: /pulsar/data/bookkeeper/journal
autoRecoveryDaemonEnabled: "false"
dbStorage_readAheadCacheMaxSizeMb: "512"
dbStorage_rocksDB_blockCacheSize: "214748364"
dbStorage_rocksDB_writeBufferSizeMB: "64"
dbStorage_writeCacheMaxSizeMb: "512"
fileInfoFormatVersionToWrite: "1"
httpServerEnabled: "true"
httpServerPort: "8000"
journalDirectories: /pulsar/data/bookkeeper/journal
journalFormatVersionToWrite: "6"
journalMaxBackups: "1"
ledgerDirectories: /pulsar/data/bookkeeper/ledgers
openLedgerRereplicationGracePeriod: "30"
rereplicationEntryBatchSize: "5000"
statsProviderClass: org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
useHostNameAsBookieID: "true"
zkLedgersRootPath: /ledgers
zkServers: pulsar-zookeeper:2181
K8S metrics give me the CPU/ RAM comsumption of my 3 bookies pods :
> pulsar-bookie-0 42m 8403Mi
pulsar-bookie-1 51m 15649Mi
pulsar-bookie-2 49m 6004Mi
Pulsar-grafana metrics shows different results :
I don't understand those differences, can someone enlighten me ? Is this a normal behavior ?
Regards,
Max
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Maxzouz commented on issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
Maxzouz commented on issue #11687:
URL: https://github.com/apache/pulsar/issues/11687#issuecomment-905394448
Thank you.
We were planning on using 2.7.2 for now but I will submit the idea of using 2.7.3 images in our chart.
> What do you mean with "stateful YAML"? Your values.yaml passed to helm install/upgrade with --values/-f ?
Yes, sorry my english is not that great. That's exactly what I meant.
I'll update the values to add memory limits on each pulsar component and then upgrade the configuration with 'helm upgrade'
Issue can be closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11687:
URL: https://github.com/apache/pulsar/issues/11687#issuecomment-905242842
> As far as I know there is no helm charts for 2.7.3 it stops at 2.7.2 right now or maybe I'm not looking where I should be :
> https://github.com/apache/pulsar-helm-chart/tags
Yes that's a gap. The reason for that the intention has been to upgrade to Pulsar 2.8.0, but that has some breaking changes and it has been postponed. Pulsar 2.7.3 just hasn't been updated to the Helm chart in the mean time, although it would make sense.
It's possible to use the 2.7.2 Helm chart and define the images to use 2.7.3 .
One way is to specify the tags in your custom values.yaml:
```
images:
zookeeper:
tag: 2.7.3
bookie:
tag: 2.7.3
autorecovery:
tag: 2.7.3
broker:
tag: 2.7.3
proxy:
tag: 2.7.3
functions:
tag: 2.7.3
pulsar_metadata:
image:
tag: 2.7.3
```
> Regarding the JVM point, I totally agree, it's just that the memory consumption looks enormous, almost 16Go used in one pod.
That's probably an issue. Please retest with Pulsar 2.7.3 to see if it can be reproduced.
> I guess the only choice would be to setup memory limits inside the statefull YAML.
What do you mean with "stateful YAML"? Your values.yaml passed to helm install/upgrade with `--values`/`-f` ?
Yes, it's necessary to tune a lot of Pulsar parameters for production usage. The gap is that the documentation doesn't include detailed instructions about doing so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11687:
URL: https://github.com/apache/pulsar/issues/11687#issuecomment-905227310
> Pulsar version: 2.6.1
@Maxzouz Pulsar 2.6.x isn't actively maintained. Please switch to use Pulsar 2.7.3. .
> RAM usage of pulsar bookie pods shouldn't exceeds xms/xmx values
> Is this a normal behavior ?
It is expected behaviour that Pulsar bookie pods consume more memory than the `-Xmx4` and `-XX:MaxDirectMemorySize` values. Please find more information in https://stackoverflow.com/a/53624438 about how Java processes consume memory.
That also applies to Pulsar Brokers. For Bookies, there's in addition the memory allocated by RocksDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Maxzouz commented on issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
Maxzouz commented on issue #11687:
URL: https://github.com/apache/pulsar/issues/11687#issuecomment-905234654
Hello,
Thanks for your reply.
Regarding de version, the upgrade process is on the way ;).
As far as I know there is no helm charts for 2.7.3 it stops at 2.7.2 right now or maybe I'm not looking where I should be :
https://github.com/apache/pulsar-helm-chart/tags
Regarding the JVM point, I totally agree, it's just that the memory consumption looks enormous, almost 16Go used in one pod.
I guess the only choice would be to setup memory limits inside the statefull YAML.
Thank you for your time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Maxzouz edited a comment on issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
Maxzouz edited a comment on issue #11687:
URL: https://github.com/apache/pulsar/issues/11687#issuecomment-905234654
Hello,
Thanks for your reply.
Regarding the version, the upgrade process is on the way ;).
As far as I know there is no helm charts for 2.7.3 it stops at 2.7.2 right now or maybe I'm not looking where I should be :
https://github.com/apache/pulsar-helm-chart/tags
Regarding the JVM point, I totally agree, it's just that the memory consumption looks enormous, almost 16Go used in one pod.
I guess the only choice would be to setup memory limits inside the statefull YAML.
Thank you for your time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Maxzouz closed issue #11687: Excessive RAM consumption on kubernetes pulsar bookie pods
Posted by GitBox <gi...@apache.org>.
Maxzouz closed issue #11687:
URL: https://github.com/apache/pulsar/issues/11687
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org