Re: SOLR Performance on RHEL 7

[Elaine Cario <et...@gmail.com>]

I'm late to the dance but FWIW, we also experienced some similar swap-like
issues when we upgraded from Centos 7.6 to Centos 7.9 (this was Solr 8.3) -
some of the Solr nodes would end up reading from disk like crazy, and query
response times would suffer accordingly.  At one point we had 1/2 the nodes
(with 1 set of replicas) on 7.6 and the other 1/2 (with 2nd replicas) on
7.9, and could see disk reads and io waits an order of magnitude higher on
7.9, with all other things being equal.

We never really solved it: after countless weeks of testing
various configurations, we threw up our hands and started migrating
everything to Amazon Linux 2 (there were other reasons for that, but this
was a definite driver).  We also have some servers still hosting RedHat
Enterprise 7.9 so far without issues but these are also slated for
migration in the coming weeks.

On Tue, Oct 26, 2021 at 8:11 AM Paul Russell <pa...@qflow.com> wrote:

> I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All SOLR
> instances use a 25G JVM on the RHEL 6 server configured with 64G of memory
> managing a 900G collection. Measured response time to queries average about
> 100ms.
>
> I am attempting to move the cluster to new RHEL 7 servers with the same
> configuration (8 cores/ 64G memory) and having performance issues.
>
> On the RHEL 7 servers the kswapd0 process is consuming up to 30% of the CPU
> and response time is being measured at 500-1000 ms for queries.
>
> I tried using the vm.swappiness setting at both 0 and 1 and have been
> unable to change the behavior. If I trim the SOLR JVM to 16Gb response
> times get better and GC logs show the JVM is operating correctly..
>
> Has anyone else had a similar issue? I have tried upgrading to SOLR 7.7.2
> as part of the process and that hasn't helped.
>
> Any suggestions?
>

Re: SOLR Performance on RHEL 7

[Paul Russell <pa...@qflow.com>]

Elaine,

Thanks for the feedback. That's troubling to hear that your issues with
SOLR and RHEL 7 weren' resolved..... Perhaps I should pursue an alternate
O/S as well if needed

I finally got the customer to add the 64Gb SDD swap volume. Strange thing
is with both the O/S tools and a Grafana node exporter running I don't see
any usage of swap at all. Because all storage was switched to SSD and I
just brought the RHEL 7 node into the cluster and I don't know if I have a
fix yet.

I appreciate the similarities.. Tomorrow will tell the tale pretty quickly.

On Wed, Dec 8, 2021 at 6:13 PM Elaine Cario <et...@gmail.com> wrote:

> I'm late to the dance but FWIW, we also experienced some similar swap-like
> issues when we upgraded from Centos 7.6 to Centos 7.9 (this was Solr 8.3) -
> some of the Solr nodes would end up reading from disk like crazy, and query
> response times would suffer accordingly.  At one point we had 1/2 the nodes
> (with 1 set of replicas) on 7.6 and the other 1/2 (with 2nd replicas) on
> 7.9, and could see disk reads and io waits an order of magnitude higher on
> 7.9, with all other things being equal.
>
> We never really solved it: after countless weeks of testing
> various configurations, we threw up our hands and started migrating
> everything to Amazon Linux 2 (there were other reasons for that, but this
> was a definite driver).  We also have some servers still hosting RedHat
> Enterprise 7.9 so far without issues but these are also slated for
> migration in the coming weeks.
>
> On Tue, Oct 26, 2021 at 8:11 AM Paul Russell <pa...@qflow.com>
> wrote:
>
> > I have a current SOLR cluster running SOLR 6.6 on RHEL 6 servers. All
> SOLR
> > instances use a 25G JVM on the RHEL 6 server configured with 64G of
> memory
> > managing a 900G collection. Measured response time to queries average
> about
> > 100ms.
> >
> > I am attempting to move the cluster to new RHEL 7 servers with the same
> > configuration (8 cores/ 64G memory) and having performance issues.
> >
> > On the RHEL 7 servers the kswapd0 process is consuming up to 30% of the
> CPU
> > and response time is being measured at 500-1000 ms for queries.
> >
> > I tried using the vm.swappiness setting at both 0 and 1 and have been
> > unable to change the behavior. If I trim the SOLR JVM to 16Gb response
> > times get better and GC logs show the JVM is operating correctly..
> >
> > Has anyone else had a similar issue? I have tried upgrading to SOLR 7.7.2
> > as part of the process and that hasn't helped.
> >
> > Any suggestions?
> >
>