You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Satya Nand <sa...@indiamart.com.INVALID> on 2022/05/05 07:01:09 UTC
Re: High CPU utilisation on Solr-8.11.0
Hi Michael,
1. set `pf=` (phrase field empty string), disabling implicit phrase query
> building. This would help give a sense of whether phrase queries are
> involved in the performance issues you're seeing.
We are also in the process of moving from standalone 6.6 to 8.7 Solr cloud,
We also noticed a huge response time increase (91 ms To 170 ms +).
We tried applying the tweak of disabling the pf field and response time was
back to normal. So somehow pf was responsible for increased response time.
We are using both query-time multi-term synonyms and
WordDelimiter[Graph]Filter.
What should we do next from here as we can't disable the pf field?
Cluster Configuration:
3 Solr Nodes: 5 CPU, 42 GB Ram (Each)
3 Zookeeper Nodes: 1 CPU, 2 GB Ram (Each)
3 Shards: 42m Documents, 42 GB (Each)
Heap: 8 GB
There are no deleted documents in the cluster and no updates going on. We
are trying to match the performance first.
On Sat, Mar 26, 2022 at 9:42 PM Michael Gibney <mi...@michaelgibney.net>
wrote:
> Are you using query-time multi-term synonyms or WordDelimiter[Graph]Filter?
> -- these can trigger "graph phrase" queries, which are handled _quite_
> differently in Solr 8.11 vs 6.5 (and although unlikely to directly cause
> the performance issues you're observing, might well explain the performance
> discrepancy). If you're _not_ using either of those, then the rest of this
> message is likely irrelevant.
>
> One thing to possibly keep an eye out for (in addition to gathering more
> evidence, as Mike Drob suggests): 6.5 started using span queries for "graph
> phrase" queries (LUCENE-7699), but the resulting phrase queries were
> completely ignored in Solr (bug) until 7.6 (SOLR-12243). Completely
> ignoring complex phrase queries did however greatly reduce latency and CPU
> load on 6.5!
>
> 7.6 started paying attention to these queries again (SOLR-12243), but also
> went back to "fully-enumerated" combinatoric approach to phrase queries
> when `ps` (phrase slop) is greater than 0 (LUCENE-8531).
>
> Some parameters you could tweak, assuming you're using edismax:
> 1. set `pf=` (phrase field empty string), disabling implicit phrase query
> building. This would help give a sense of whether phrase queries are
> involved in the performance issues you're seeing.
> 2. set `ps=0` (phrase slop 0), this should allow span queries to be built,
> which should generally be more efficient than analogous non-span-query
> approach (basically this would make the change introduced by LUCENE-8531
> irrelevant); tangentially: the special case building span queries for
> `ps=0` is removed as of Lucene 9.0 (will be removed as of Solr 9.0 -- not
> directly relevant to this issue though).
>
> Michael
>
> On Sat, Mar 26, 2022 at 8:26 AM Mike Drob <md...@mdrob.com> wrote:
>
> > Can you provide more details on what they CPU time is spent on? Maybe
> look
> > at some JFR profiles or collect several jstacks to see where they
> > bottlenecks are.
> >
> > On Sat, Mar 26, 2022 at 3:49 AM Modassar Ather <mo...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We are trying to migrate to Solr-8.11.0 from Solr-6.5.1. Following are
> > the
> > > details of Solr installation.
> > >
> > > Server : EC2 instance with 32 CPUs and 521 GB
> > > Storage : EBS Volume. General Purpose SSD (gp3) with 3000/5000 IOPS
> > > Ubuntu :Ubuntu 20.04.3 LTS
> > > Java : openjdk 11.0.14
> > > SolrCloud : 12 shards having a total 4+ TB index. Each node has a 30GB
> > max
> > > memory limit.
> > > GC setting on Solr : G1GC
> > > Solr query timeout : 5 minutes
> > >
> > > During testing we observed a high CPU utilisation and few of the
> queries
> > > with wildcard queries are timing out. These queries are getting
> executed
> > > completely on Solr-6.5.1.
> > > After tuning a few of the parameters of GC settings the CPU utilisation
> > > came down but it is still high when compared with Solr-6.5.1 and some
> > > queries with wildcard queries are still failing.
> > >
> > > Kindly provide your suggestions.
> > >
> > > Thanks,
> > > Modassar
> > >
> >
>
Re: High CPU utilisation on Solr-8.11.0
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Thanks, Michael.
I missed that mail thread among all responses. I will check that too.
On Thu, May 5, 2022 at 6:26 PM Michael Gibney <mi...@michaelgibney.net>
wrote:
> Did you yet dig through the mailing list thread link that I posted earlier
> in this thread? It explains in more depth, suggests a number of possible
> mitigations, and has a bunch of links to jira issues that provide extra
> context. Off the cuff, I'd say that setting `enableGraphQueries=false` may
> be most immediately helpful in terms of restoring performance.
>
> (As an aside: from my perspective though, even if you can restore
> performance, it would be at the expense of nuances of functionality. Longer
> term I'd really like to help solve this properly, involving some
> combination of the issues linked to in the above thread ...)
>
> Michael
>
> On Thu, May 5, 2022 at 3:01 AM Satya Nand <satya.nand@indiamart.com
> .invalid>
> wrote:
>
> > Hi Michael,
> >
> >
> > 1. set `pf=` (phrase field empty string), disabling implicit phrase query
> > > building. This would help give a sense of whether phrase queries are
> > > involved in the performance issues you're seeing.
> >
> >
> > We are also in the process of moving from standalone 6.6 to 8.7 Solr
> cloud,
> > We also noticed a huge response time increase (91 ms To 170 ms +).
> >
> > We tried applying the tweak of disabling the pf field and response time
> was
> > back to normal. So somehow pf was responsible for increased response
> time.
> >
> > We are using both query-time multi-term synonyms and
> > WordDelimiter[Graph]Filter.
> >
> > What should we do next from here as we can't disable the pf field?
> >
> > Cluster Configuration:
> >
> > 3 Solr Nodes: 5 CPU, 42 GB Ram (Each)
> > 3 Zookeeper Nodes: 1 CPU, 2 GB Ram (Each)
> > 3 Shards: 42m Documents, 42 GB (Each)
> > Heap: 8 GB
> >
> >
> > There are no deleted documents in the cluster and no updates going on. We
> > are trying to match the performance first.
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Mar 26, 2022 at 9:42 PM Michael Gibney <
> michael@michaelgibney.net>
> > wrote:
> >
> > > Are you using query-time multi-term synonyms or
> > WordDelimiter[Graph]Filter?
> > > -- these can trigger "graph phrase" queries, which are handled _quite_
> > > differently in Solr 8.11 vs 6.5 (and although unlikely to directly
> cause
> > > the performance issues you're observing, might well explain the
> > performance
> > > discrepancy). If you're _not_ using either of those, then the rest of
> > this
> > > message is likely irrelevant.
> > >
> > > One thing to possibly keep an eye out for (in addition to gathering
> more
> > > evidence, as Mike Drob suggests): 6.5 started using span queries for
> > "graph
> > > phrase" queries (LUCENE-7699), but the resulting phrase queries were
> > > completely ignored in Solr (bug) until 7.6 (SOLR-12243). Completely
> > > ignoring complex phrase queries did however greatly reduce latency and
> > CPU
> > > load on 6.5!
> > >
> > > 7.6 started paying attention to these queries again (SOLR-12243), but
> > also
> > > went back to "fully-enumerated" combinatoric approach to phrase queries
> > > when `ps` (phrase slop) is greater than 0 (LUCENE-8531).
> > >
> > > Some parameters you could tweak, assuming you're using edismax:
> > > 1. set `pf=` (phrase field empty string), disabling implicit phrase
> query
> > > building. This would help give a sense of whether phrase queries are
> > > involved in the performance issues you're seeing.
> > > 2. set `ps=0` (phrase slop 0), this should allow span queries to be
> > built,
> > > which should generally be more efficient than analogous non-span-query
> > > approach (basically this would make the change introduced by
> LUCENE-8531
> > > irrelevant); tangentially: the special case building span queries for
> > > `ps=0` is removed as of Lucene 9.0 (will be removed as of Solr 9.0 --
> not
> > > directly relevant to this issue though).
> > >
> > > Michael
> > >
> > > On Sat, Mar 26, 2022 at 8:26 AM Mike Drob <md...@mdrob.com> wrote:
> > >
> > > > Can you provide more details on what they CPU time is spent on? Maybe
> > > look
> > > > at some JFR profiles or collect several jstacks to see where they
> > > > bottlenecks are.
> > > >
> > > > On Sat, Mar 26, 2022 at 3:49 AM Modassar Ather <
> modather1981@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are trying to migrate to Solr-8.11.0 from Solr-6.5.1. Following
> > are
> > > > the
> > > > > details of Solr installation.
> > > > >
> > > > > Server : EC2 instance with 32 CPUs and 521 GB
> > > > > Storage : EBS Volume. General Purpose SSD (gp3) with 3000/5000 IOPS
> > > > > Ubuntu :Ubuntu 20.04.3 LTS
> > > > > Java : openjdk 11.0.14
> > > > > SolrCloud : 12 shards having a total 4+ TB index. Each node has a
> > 30GB
> > > > max
> > > > > memory limit.
> > > > > GC setting on Solr : G1GC
> > > > > Solr query timeout : 5 minutes
> > > > >
> > > > > During testing we observed a high CPU utilisation and few of the
> > > queries
> > > > > with wildcard queries are timing out. These queries are getting
> > > executed
> > > > > completely on Solr-6.5.1.
> > > > > After tuning a few of the parameters of GC settings the CPU
> > utilisation
> > > > > came down but it is still high when compared with Solr-6.5.1 and
> some
> > > > > queries with wildcard queries are still failing.
> > > > >
> > > > > Kindly provide your suggestions.
> > > > >
> > > > > Thanks,
> > > > > Modassar
> > > > >
> > > >
> > >
> >
>
Re: High CPU utilisation on Solr-8.11.0
Posted by Michael Gibney <mi...@michaelgibney.net>.
Did you yet dig through the mailing list thread link that I posted earlier
in this thread? It explains in more depth, suggests a number of possible
mitigations, and has a bunch of links to jira issues that provide extra
context. Off the cuff, I'd say that setting `enableGraphQueries=false` may
be most immediately helpful in terms of restoring performance.
(As an aside: from my perspective though, even if you can restore
performance, it would be at the expense of nuances of functionality. Longer
term I'd really like to help solve this properly, involving some
combination of the issues linked to in the above thread ...)
Michael
On Thu, May 5, 2022 at 3:01 AM Satya Nand <sa...@indiamart.com.invalid>
wrote:
> Hi Michael,
>
>
> 1. set `pf=` (phrase field empty string), disabling implicit phrase query
> > building. This would help give a sense of whether phrase queries are
> > involved in the performance issues you're seeing.
>
>
> We are also in the process of moving from standalone 6.6 to 8.7 Solr cloud,
> We also noticed a huge response time increase (91 ms To 170 ms +).
>
> We tried applying the tweak of disabling the pf field and response time was
> back to normal. So somehow pf was responsible for increased response time.
>
> We are using both query-time multi-term synonyms and
> WordDelimiter[Graph]Filter.
>
> What should we do next from here as we can't disable the pf field?
>
> Cluster Configuration:
>
> 3 Solr Nodes: 5 CPU, 42 GB Ram (Each)
> 3 Zookeeper Nodes: 1 CPU, 2 GB Ram (Each)
> 3 Shards: 42m Documents, 42 GB (Each)
> Heap: 8 GB
>
>
> There are no deleted documents in the cluster and no updates going on. We
> are trying to match the performance first.
>
>
>
>
>
>
>
> On Sat, Mar 26, 2022 at 9:42 PM Michael Gibney <mi...@michaelgibney.net>
> wrote:
>
> > Are you using query-time multi-term synonyms or
> WordDelimiter[Graph]Filter?
> > -- these can trigger "graph phrase" queries, which are handled _quite_
> > differently in Solr 8.11 vs 6.5 (and although unlikely to directly cause
> > the performance issues you're observing, might well explain the
> performance
> > discrepancy). If you're _not_ using either of those, then the rest of
> this
> > message is likely irrelevant.
> >
> > One thing to possibly keep an eye out for (in addition to gathering more
> > evidence, as Mike Drob suggests): 6.5 started using span queries for
> "graph
> > phrase" queries (LUCENE-7699), but the resulting phrase queries were
> > completely ignored in Solr (bug) until 7.6 (SOLR-12243). Completely
> > ignoring complex phrase queries did however greatly reduce latency and
> CPU
> > load on 6.5!
> >
> > 7.6 started paying attention to these queries again (SOLR-12243), but
> also
> > went back to "fully-enumerated" combinatoric approach to phrase queries
> > when `ps` (phrase slop) is greater than 0 (LUCENE-8531).
> >
> > Some parameters you could tweak, assuming you're using edismax:
> > 1. set `pf=` (phrase field empty string), disabling implicit phrase query
> > building. This would help give a sense of whether phrase queries are
> > involved in the performance issues you're seeing.
> > 2. set `ps=0` (phrase slop 0), this should allow span queries to be
> built,
> > which should generally be more efficient than analogous non-span-query
> > approach (basically this would make the change introduced by LUCENE-8531
> > irrelevant); tangentially: the special case building span queries for
> > `ps=0` is removed as of Lucene 9.0 (will be removed as of Solr 9.0 -- not
> > directly relevant to this issue though).
> >
> > Michael
> >
> > On Sat, Mar 26, 2022 at 8:26 AM Mike Drob <md...@mdrob.com> wrote:
> >
> > > Can you provide more details on what they CPU time is spent on? Maybe
> > look
> > > at some JFR profiles or collect several jstacks to see where they
> > > bottlenecks are.
> > >
> > > On Sat, Mar 26, 2022 at 3:49 AM Modassar Ather <modather1981@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are trying to migrate to Solr-8.11.0 from Solr-6.5.1. Following
> are
> > > the
> > > > details of Solr installation.
> > > >
> > > > Server : EC2 instance with 32 CPUs and 521 GB
> > > > Storage : EBS Volume. General Purpose SSD (gp3) with 3000/5000 IOPS
> > > > Ubuntu :Ubuntu 20.04.3 LTS
> > > > Java : openjdk 11.0.14
> > > > SolrCloud : 12 shards having a total 4+ TB index. Each node has a
> 30GB
> > > max
> > > > memory limit.
> > > > GC setting on Solr : G1GC
> > > > Solr query timeout : 5 minutes
> > > >
> > > > During testing we observed a high CPU utilisation and few of the
> > queries
> > > > with wildcard queries are timing out. These queries are getting
> > executed
> > > > completely on Solr-6.5.1.
> > > > After tuning a few of the parameters of GC settings the CPU
> utilisation
> > > > came down but it is still high when compared with Solr-6.5.1 and some
> > > > queries with wildcard queries are still failing.
> > > >
> > > > Kindly provide your suggestions.
> > > >
> > > > Thanks,
> > > > Modassar
> > > >
> > >
> >
>