You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2022/01/06 21:03:42 UTC
Propose Solr 9 *Docker* image use Java 17
I'd like to propose that our Docker image for Solr 9 move from Java 11 to
Java 17. Admittedly I don't have any familiarity with running 17, so I
would really like to hear from those of you using it. I'm guessing
(informed from some quick google searches) there are some ~minor
performance improvements but nothing eye-popping there. Mostly, I propose
this because a 9.0 release is an ideal time to make such a change instead
of some minor release in between that could introduce a subtle surprise for
some users. The new Shenandoah GC looks exciting but may not be
sufficiently ready for us to recommend (if I recall from a recent user who
reported a problem with it) -- and that's okay. Having this as an option
for users is great, especially as time progresses and future Docker Solr
releases include minor updates to the JVM base image that will increase the
viability.
I'm aware our nifty image building enables people to do a custom build to
specify their own preferred FROM image, which is cool. Still, I think we
should move on to 17 as the default.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Re: Propose Solr 9 *Docker* image use Java 17
Posted by David Smiley <ds...@apache.org>.
And to those of you who may not know, our Docker Solr image for Solr 8 uses
Java 11 even though Solr 8 supports Java 8. Solr 9 increases to require
Java 11 (not Java 17) and I'm proposing only bumping the Docker-Solr
default accordingly upwards (newer). In a container-ized world, I think
picking the most recent LTS (which is currently Java 17) should be our
standard practice because the onus on upgrading is on *us*, unlike classic
bare metal where upgrading effort is on the user. Users have ask for this:
https://github.com/docker-solr/docker-solr/issues/231 (3 people +1'ed my
proposal to move to Java 17 at this juncture)
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Jan 6, 2022 at 4:03 PM David Smiley <ds...@apache.org> wrote:
> I'd like to propose that our Docker image for Solr 9 move from Java 11 to
> Java 17. Admittedly I don't have any familiarity with running 17, so I
> would really like to hear from those of you using it. I'm guessing
> (informed from some quick google searches) there are some ~minor
> performance improvements but nothing eye-popping there. Mostly, I propose
> this because a 9.0 release is an ideal time to make such a change instead
> of some minor release in between that could introduce a subtle surprise for
> some users. The new Shenandoah GC looks exciting but may not be
> sufficiently ready for us to recommend (if I recall from a recent user who
> reported a problem with it) -- and that's okay. Having this as an option
> for users is great, especially as time progresses and future Docker Solr
> releases include minor updates to the JVM base image that will increase the
> viability.
>
> I'm aware our nifty image building enables people to do a custom build to
> specify their own preferred FROM image, which is cool. Still, I think we
> should move on to 17 as the default.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Re: Propose Solr 9 *Docker* image use Java 17
Posted by Houston Putman <ho...@apache.org>.
I'm fine switching to java 17, but it would be nice to see some benchmarks
first before making it the default.
> Actually, it would be nice if we could publish all our images under apache
> name-space, and then have docker folks symlink /_/solr to these like they
> do for elastic.
>
We were explicitly told by the Docker folks that they are not going to
green light anything like this going forward, and they wished they hadn't
done it for elastic.
So definitely a no-go.
Or we could keep the official image on Java 11 to be conservative, and at
> the same time publish some variants of our own under the apache namespace:
This would certainly be possible and easy to do.
I don' tknow if we need different Dockerfile.body templates for the
> variants or if they could be built with just different build-args for FROM?
We could either use the local or official docker file that is generated for
the release, it doesn't really matter. All we would have to do is change
the build-arg for the FROM.
- Houston
On Thu, Jan 6, 2022 at 4:27 PM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Definitely a possibility.
> Or we could keep the official image on Java 11 to be conservative, and at
> the same time publish some variants of our own under the apache namespace:
>
> apache/solr:9.0.0-jre17
> apache/solr:9.0.0-jdk17
> apache/solr:9.0.0-jre11
> apache/solr:9.0.0-jdk11
>
> I don' tknow if we need different Dockerfile.body templates for the
> variants or if they could be built with just different build-args for FROM?
>
> Actually, it would be nice if we could publish all our images under apache
> name-space, and then have docker folks symlink /_/solr to these like they
> do for elastic. It would give us more freedom with advanced multi step
> builds and smaller images. But I think that door is closed now.
>
> Jan
>
> 6. jan. 2022 kl. 22:03 skrev David Smiley <ds...@apache.org>:
>
> I'd like to propose that our Docker image for Solr 9 move from Java 11 to
> Java 17. Admittedly I don't have any familiarity with running 17, so I
> would really like to hear from those of you using it. I'm guessing
> (informed from some quick google searches) there are some ~minor
> performance improvements but nothing eye-popping there. Mostly, I propose
> this because a 9.0 release is an ideal time to make such a change instead
> of some minor release in between that could introduce a subtle surprise for
> some users. The new Shenandoah GC looks exciting but may not be
> sufficiently ready for us to recommend (if I recall from a recent user who
> reported a problem with it) -- and that's okay. Having this as an option
> for users is great, especially as time progresses and future Docker Solr
> releases include minor updates to the JVM base image that will increase the
> viability.
>
> I'm aware our nifty image building enables people to do a custom build to
> specify their own preferred FROM image, which is cool. Still, I think we
> should move on to 17 as the default.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
>
Re: Propose Solr 9 *Docker* image use Java 17
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Definitely a possibility.
Or we could keep the official image on Java 11 to be conservative, and at the same time publish some variants of our own under the apache namespace:
apache/solr:9.0.0-jre17
apache/solr:9.0.0-jdk17
apache/solr:9.0.0-jre11
apache/solr:9.0.0-jdk11
I don' tknow if we need different Dockerfile.body templates for the variants or if they could be built with just different build-args for FROM?
Actually, it would be nice if we could publish all our images under apache name-space, and then have docker folks symlink /_/solr to these like they do for elastic. It would give us more freedom with advanced multi step builds and smaller images. But I think that door is closed now.
Jan
> 6. jan. 2022 kl. 22:03 skrev David Smiley <ds...@apache.org>:
>
> I'd like to propose that our Docker image for Solr 9 move from Java 11 to Java 17. Admittedly I don't have any familiarity with running 17, so I would really like to hear from those of you using it. I'm guessing (informed from some quick google searches) there are some ~minor performance improvements but nothing eye-popping there. Mostly, I propose this because a 9.0 release is an ideal time to make such a change instead of some minor release in between that could introduce a subtle surprise for some users. The new Shenandoah GC looks exciting but may not be sufficiently ready for us to recommend (if I recall from a recent user who reported a problem with it) -- and that's okay. Having this as an option for users is great, especially as time progresses and future Docker Solr releases include minor updates to the JVM base image that will increase the viability.
>
> I'm aware our nifty image building enables people to do a custom build to specify their own preferred FROM image, which is cool. Still, I think we should move on to 17 as the default.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
Re: Propose Solr 9 *Docker* image use Java 17
Posted by David Smiley <ds...@apache.org>.
Closing the loop here: https://issues.apache.org/jira/browse/SOLR-15949 for
Java 17 via the Eclipse Temurin distribution. I'll merge this weekend.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Sun, Jan 9, 2022 at 3:01 AM Mark Miller <ma...@gmail.com> wrote:
> That is generally an expected trade off when using the new collectors -
> perhaps a 15% hit to throughput due to locking in concurrent compaction but
> better latency with the shorter pauses.
>
> The last I saw, for smaller heaps, G1 generally wins - better
> throughput and latency that’s just as good.
>
> For larger heaps, if memory is not a constraint, the new collectors win.
>
> If memory is a constraint, you pay for the better latency with throughput
> with the new collectors.
>
> G1 remains a good default generally.
>
>
> *Mark Miller* - Chat @ Spike
> <https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=1deyla> [image:
> 1deyla]
>
> On January 8, 2022 at 2:33 GMT, Shawn Heisey <ap...@elyograg.org> wrote:
>
>
> On 1/6/2022 2:03 PM, David Smiley wrote:
> > The new Shenandoah GC looks exciting but may not be sufficiently ready
> > for us to recommend (if I recall from a recent user who reported a
> > problem with it) -- and that's okay.
>
> I've done some experiments with Shenandoah and ZGC. My index is tiny,
> 155297 docs and a total index size of 644.22MB. I'm running it with a
> max heap of 512MB.
>
> When I uploaded the GC logs to gceasy, the newer collectors showed much
> smaller pauses than G1, and more collections. I think the overall pause
> time is significantly smaller, but throughput took a hit. It was a
> larger impact than I imagined. Last time I tested it, a full rebuild of
> the index took about 8 minutes with G1, and over 9 minutes with the
> newer collectors. That index is built and used by my dovecot install.
> There's pretty much no query activity on my system, so I used the full
> index rebuild to exercise it.
>
> I had a remote co-conspirator on this testing. On that user's index,
> their reindex procedure failed to complete at all with the newer
> collectors, but it did work with G1. I did not get any details about
> how it failed, but I know that their testing and mine were done with
> OpenJDK 11.
>
> I was going to do some additional testing with OpenJDK 17.0.1, but I
> found that when I tried to start Solr with Shenandoah, it has the
> following in the console log:
>
> Error occurred during initialization of VM
> Option -XX:+UseShenandoahGC not supported
>
> This is very weird, because I saw an article about sub-millisecond
> pauses using OpenJDK 17 and Shenandoah. Unless Oracle has decided to
> pull Shenandoah completely in-house and not make it available in OpenJDK
> any more.
>
> Thanks,
> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
>
>
Re: Propose Solr 9 *Docker* image use Java 17
Posted by Mark Miller <ma...@gmail.com>.
That is generally an expected trade off when using the new collectors - perhaps a 15% hit to throughput due to locking in concurrent compaction but better latency with the shorter pauses.
The last I saw, for smaller heaps, G1 generally wins - better throughput and latency that’s just as good.
For larger heaps, if memory is not a constraint, the new collectors win.
If memory is a constraint, you pay for the better latency with throughput with the new collectors.
G1 remains a good default generally.
[Mark Miller - Chat @ Spike](https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=1deyla) [1deyla]
On January 8, 2022 at 2:33 GMT, Shawn Heisey <ap...@elyograg.org> wrote:
On 1/6/2022 2:03 PM, David Smiley wrote:
> The new Shenandoah GC looks exciting but may not be sufficiently ready
> for us to recommend (if I recall from a recent user who reported a
> problem with it) -- and that's okay.
I've done some experiments with Shenandoah and ZGC. My index is tiny,
155297 docs and a total index size of 644.22MB. I'm running it with a
max heap of 512MB.
When I uploaded the GC logs to gceasy, the newer collectors showed much
smaller pauses than G1, and more collections. I think the overall pause
time is significantly smaller, but throughput took a hit. It was a
larger impact than I imagined. Last time I tested it, a full rebuild of
the index took about 8 minutes with G1, and over 9 minutes with the
newer collectors. That index is built and used by my dovecot install.
There's pretty much no query activity on my system, so I used the full
index rebuild to exercise it.
I had a remote co-conspirator on this testing. On that user's index,
their reindex procedure failed to complete at all with the newer
collectors, but it did work with G1. I did not get any details about
how it failed, but I know that their testing and mine were done with
OpenJDK 11.
I was going to do some additional testing with OpenJDK 17.0.1, but I
found that when I tried to start Solr with Shenandoah, it has the
following in the console log:
Error occurred during initialization of VM
Option -XX:+UseShenandoahGC not supported
This is very weird, because I saw an article about sub-millisecond
pauses using OpenJDK 17 and Shenandoah. Unless Oracle has decided to
pull Shenandoah completely in-house and not make it available in OpenJDK
any more.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Propose Solr 9 *Docker* image use Java 17
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/6/2022 2:03 PM, David Smiley wrote:
> The new Shenandoah GC looks exciting but may not be sufficiently ready
> for us to recommend (if I recall from a recent user who reported a
> problem with it) -- and that's okay.
I've done some experiments with Shenandoah and ZGC. My index is tiny,
155297 docs and a total index size of 644.22MB. I'm running it with a
max heap of 512MB.
When I uploaded the GC logs to gceasy, the newer collectors showed much
smaller pauses than G1, and more collections. I think the overall pause
time is significantly smaller, but throughput took a hit. It was a
larger impact than I imagined. Last time I tested it, a full rebuild of
the index took about 8 minutes with G1, and over 9 minutes with the
newer collectors. That index is built and used by my dovecot install.
There's pretty much no query activity on my system, so I used the full
index rebuild to exercise it.
I had a remote co-conspirator on this testing. On that user's index,
their reindex procedure failed to complete at all with the newer
collectors, but it did work with G1. I did not get any details about
how it failed, but I know that their testing and mine were done with
OpenJDK 11.
I was going to do some additional testing with OpenJDK 17.0.1, but I
found that when I tried to start Solr with Shenandoah, it has the
following in the console log:
Error occurred during initialization of VM
Option -XX:+UseShenandoahGC not supported
This is very weird, because I saw an article about sub-millisecond
pauses using OpenJDK 17 and Shenandoah. Unless Oracle has decided to
pull Shenandoah completely in-house and not make it available in OpenJDK
any more.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org