You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/20 08:17:44 UTC
[GitHub] [pulsar] Skaronator opened a new issue #7859: Use Java 11 for Docker Images
Skaronator opened a new issue #7859:
URL: https://github.com/apache/pulsar/issues/7859
**Is your enhancement request related to a problem? Please describe.**
The current docker images are built with Java 8. According to the [2.3.0 - 2019-02-20](https://pulsar.apache.org/release-notes/#230-mdash-2019-02-20-a-id230a) changelog Pulsar supports Java 11 since version
Java 8 and Docker doesn't work that well together. Some things got backported to Java 8 but only Java 10+ supports Memory and CPU limits inside Docker Containers correctly. More on that here: https://www.royvanrijn.com/blog/2018/05/java-and-docker-memory-limits/
**Describe the solution you'd like**
I'd like to see Docker Images that use Java 11 & use the correct start parameters by default. The current images rely on openjdk:8
https://github.com/apache/pulsar/blob/b410274554508684f4a15cc1f5a98a71d85fbe2e/docker/pulsar/Dockerfile#L41
https://github.com/apache/pulsar/blob/b410274554508684f4a15cc1f5a98a71d85fbe2e/docker/pulsar-standalone/Dockerfile#L27
**Describe alternatives you've considered**
Alternatively there could be a java11 version of the image, parallel to the current images. Like "2.7.0" and "2.7.0-java11" https://hub.docker.com/r/apachepulsar/pulsar/tags
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
lhotari closed issue #7859:
URL: https://github.com/apache/pulsar/issues/7859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gvolpe commented on issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
gvolpe commented on issue #7859:
URL: https://github.com/apache/pulsar/issues/7859#issuecomment-698298490
:+1: to using JDK 11
We have run into some issues due to Pulsar using JDK 8 (quite old these days) and we are now considering creating our own Docker image for Pulsar instead.
We have some Pulsar functions being compiled with JDK 11 but they need to run on JDK 8 and there are a lot of incompatibilities.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #7859:
URL: https://github.com/apache/pulsar/issues/7859#issuecomment-686272487
> Java 8 and Docker doesn't work that well together. Some things got backported to Java 8 but only Java 10+ supports Memory and CPU limits inside Docker Containers correctly.
This has changed in the last 2 years. The Java 8 version used in Pulsar images already supports container memory & cpu limits. [This blog post explains it](https://blog.softwaremill.com/docker-support-in-new-java-8-finally-fd595df0ca54).
One way to verify that it's supported is to run this docker command:
```
docker run --memory 100m --cpus 1 --rm --entrypoint '/bin/bash' apachepulsar/pulsar:2.6.1 -c 'java -XX:+UnlockDiagnosticVMOptions -XX:+PrintContainerInfo -version'
```
it will printout diagnostics about the limits that the JVM detects:
```
OSContainer::init: Initializing Container Support
Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes
Memory Limit is: 104857600
Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
CPU Quota is: 100000
Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
CPU Period is: 100000
Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/cpu.shares
CPU Shares is: 1024
CPU Quota count based on quota/period: 1
OSContainer::active_processor_count: 1
```
It's already possible to use the JVM container support in Pulsar containers by specifying `PULSAR_MEM="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 -XX:MinRAMPercentage=50.0"`.
For example:
```
docker run --name pulsar-standalone --memory 512m --cpus 1 -e PULSAR_MEM="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 -XX:MinRAMPercentage=50.0" -e PULSAR_GC="-XX:+UseG1GC" apachepulsar/pulsar:2.6.1 bin/pulsar standalone
```
Verifying in another terminal after startup:
```
$ docker exec pulsar-standalone jcmd 1 GC.heap_info
1:
garbage-first heap total 262144K, used 146496K [0x00000000f0000000, 0x00000000f0100800, 0x0000000100000000)
region size 1024K, 75 young (76800K), 7 survivors (7168K)
Metaspace used 78433K, capacity 82292K, committed 82460K, reserved 1122304K
class space used 9461K, capacity 10287K, committed 10316K, reserved 1048576K
```
Heap size is 256MB, which is expected. Since Pulsar uses also native memory, it's probably not worth increasing the heap limit too much over 50% so that the container doesn't get killed. The [default MaxDirectMemorySize is the maximum heap size](https://stackoverflow.com/a/54447744). The [total memory consumption of a Java process is explained here](https://stackoverflow.com/a/53624438).
It's also worth noticing that PULSAR_GC should be set for docker containers so that the not-so-good defaults don't get used. [There's a PR already for addressing this issue](https://github.com/apache/pulsar/pull/3650). One issue with the defaults is that there's a huge number of GC threads configured.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gvolpe commented on issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
gvolpe commented on issue #7859:
URL: https://github.com/apache/pulsar/issues/7859#issuecomment-698298490
:+1: to using JDK 11
We have run into some issues due to Pulsar using JDK 8 (quite old these days) and we are now considering creating our own Docker image for Pulsar instead.
We have some Pulsar functions being compiled with JDK 11 but they need to run on JDK 8 and there are a lot of incompatibilities.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #7859:
URL: https://github.com/apache/pulsar/issues/7859#issuecomment-678160537
Using Java 11 for the Pulsar docker images won't solve the memory limit issue alone. By default, the Pulsar docker image uses the pulsar_env.sh script to set the memory limits, https://github.com/apache/pulsar/blob/master/conf/pulsar_env.sh#L45 . For Bookies, it's in the bkenv.sh script: https://github.com/apache/pulsar/blob/master/conf/bkenv.sh#L36 .
The default memory limits are static and are not calculated based on the available memory to the container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on issue #7859: Use Java 11 for Docker Images
Posted by GitBox <gi...@apache.org>.
cbornet commented on issue #7859:
URL: https://github.com/apache/pulsar/issues/7859#issuecomment-985366151
The images now use JDK11. I believe this issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org