You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vishal patel <vi...@outlook.com> on 2020/05/14 14:09:07 UTC
Performance issue in Query execution in Solr 8.3.0 and 8.5.1
I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1 when values of one field is large in query and group field is apply.
My Solr URL : https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
My Solr config and schema : https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn<https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn>
It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5 seconds in Solr 6.1.0.
Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
Regards,
Vishal Patel
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
Any one is looking my issue? Due to this issue I can not upgrade Solr 8.3.0.
regards,
Vishal Patel
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Sunday, May 17, 2020 11:49 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Solr 6.1.0 : <int name="QTime">1881</int>
Here is my thread dump stack trace and log for Solr 6.1.0. It is helpful for you.
My threads: qtp557041912-245356 and qtp557041912-245342.
https://drive.google.com/file/d/1owtotYEnJacMiEZyuGLk3AHQ9kQG5rww/view?usp=sharing
Regards
Vishal Patel
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Sunday, May 17, 2020 11:04 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Thanks for reply.
I know Query field value is large. But same thing is working fine in Solr 6.1.0 and query executed within 300 milliseconds. Schema.xml and Solrconfig.xml are same. Why is it taking lots of time for execution in Solr 8.3.0?
Is there any changes in Solr 8.3.0?
Regards,
Vishal Patel
________________________________
From: Mikhail Khludnev <mk...@apache.org>
Sent: Saturday, May 16, 2020 6:55 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
It seems this thread is doing heavy work, mind the bottom line.
202.8013ms
124.8008ms
qtp153245266-156 (156)
org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.<init>(BM25Similarity.java:219)
org.apache.lucene.search.similarities.BM25Similarity.scorer(BM25Similarity.java:192)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.scorer(PerFieldSimilarityWrapper.java:47)
org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:74)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:205)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:63)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:231)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.TopFieldCollector.populateScores(TopFieldCollector.java:531)
org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.postCollect(TopGroupsFieldCommand.java:178)
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:168)
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1403)
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:387)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
It seems like it ranks groups by query score, that doubtful thing to do.
From the log. Here's how to recognize query running 25 sec "QTime=25063"
Query itself q=+msg_id:(10519539+10519540+10523575+10523576+ ... is
not what search engines are made for. They are purposed for short
query.
You may
1. leverage {!terms} query parser which might handle such long terms
list more efficiently
2. make sure you don't enable unnecessary grouping features, eg group
ranking in the stack above makes no sense for this kind of query
It's worth to revamp an overall approach in favor of query time
{!join} or index time join see {!parent}/nested docs.
On Sat, May 16, 2020 at 1:46 PM vishal patel <vi...@outlook.com>
wrote:
> Thanks for reply.
>
> I have taken a thread dump at the time of query execution. I do not know
> the thread name so send the All threads. I have also send the logs so you
> can get idea.
>
> Thread Dump All Stack Trace:
> https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
> Solr 8.3 shard 1 log:
> https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
> Solr 8.3 shard 2 log:
> https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
>
> I have some questions regarding the thread dump
> - How can I know the my thread name from thread dump? can I get from the
> log?
> - When do I take a thread dump? on query execution or after query
> execution?
>
> Note: I got a thread name from log and checked in thread dump on query
> execution time and after query executed. Both time thread stack trace got
> different.
>
> If any other things are required then let me know I will send.
>
> Regards,
> Vishal Patel
> ________________________________
> From: Mikhail Khludnev <mk...@apache.org>
> Sent: Saturday, May 16, 2020 2:23 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
> 34 seconds? Please share the deepest thread stack. This might give a clue
> what's going on there.
>
> On Sat, May 16, 2020 at 11:46 AM vishal patel <
> vishalpatel200928@outlook.com>
> wrote:
>
> > Any one is looking my issue? Please help me.
> >
> > Sent from Outlook<http://aka.ms/weboutlook>
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Friday, May 15, 2020 3:06 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I have result of query debug for both version so It will helpful.
> >
> > Solr 6.1 query debug URL
> > https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> > Solr 8.3.1 query debug URL
> > https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
> >
> > I indexed same data in both version.
> >
> > I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> > of Solr 8.6.1. Is there any impact of score in query execution? why is
> > score=1.0 in result of Solr 8.3.0?
> >
> > Regards,
> > Vishal Patel
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Thursday, May 14, 2020 7:39 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
> >
> > I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> > when values of one field is large in query and group field is apply.
> >
> > My Solr URL :
> > https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> > My Solr config and schema :
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> <
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> >
> >
> > It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> > seconds in Solr 6.1.0.
> >
> > Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
> >
> >
> > Regards,
> > Vishal Patel
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
Solr 6.1.0 : <int name="QTime">1881</int>
Here is my thread dump stack trace and log for Solr 6.1.0. It is helpful for you.
My threads: qtp557041912-245356 and qtp557041912-245342.
https://drive.google.com/file/d/1owtotYEnJacMiEZyuGLk3AHQ9kQG5rww/view?usp=sharing
Regards
Vishal Patel
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Sunday, May 17, 2020 11:04 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Thanks for reply.
I know Query field value is large. But same thing is working fine in Solr 6.1.0 and query executed within 300 milliseconds. Schema.xml and Solrconfig.xml are same. Why is it taking lots of time for execution in Solr 8.3.0?
Is there any changes in Solr 8.3.0?
Regards,
Vishal Patel
________________________________
From: Mikhail Khludnev <mk...@apache.org>
Sent: Saturday, May 16, 2020 6:55 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
It seems this thread is doing heavy work, mind the bottom line.
202.8013ms
124.8008ms
qtp153245266-156 (156)
org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.<init>(BM25Similarity.java:219)
org.apache.lucene.search.similarities.BM25Similarity.scorer(BM25Similarity.java:192)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.scorer(PerFieldSimilarityWrapper.java:47)
org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:74)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:205)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:63)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:231)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.TopFieldCollector.populateScores(TopFieldCollector.java:531)
org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.postCollect(TopGroupsFieldCommand.java:178)
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:168)
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1403)
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:387)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
It seems like it ranks groups by query score, that doubtful thing to do.
From the log. Here's how to recognize query running 25 sec "QTime=25063"
Query itself q=+msg_id:(10519539+10519540+10523575+10523576+ ... is
not what search engines are made for. They are purposed for short
query.
You may
1. leverage {!terms} query parser which might handle such long terms
list more efficiently
2. make sure you don't enable unnecessary grouping features, eg group
ranking in the stack above makes no sense for this kind of query
It's worth to revamp an overall approach in favor of query time
{!join} or index time join see {!parent}/nested docs.
On Sat, May 16, 2020 at 1:46 PM vishal patel <vi...@outlook.com>
wrote:
> Thanks for reply.
>
> I have taken a thread dump at the time of query execution. I do not know
> the thread name so send the All threads. I have also send the logs so you
> can get idea.
>
> Thread Dump All Stack Trace:
> https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
> Solr 8.3 shard 1 log:
> https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
> Solr 8.3 shard 2 log:
> https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
>
> I have some questions regarding the thread dump
> - How can I know the my thread name from thread dump? can I get from the
> log?
> - When do I take a thread dump? on query execution or after query
> execution?
>
> Note: I got a thread name from log and checked in thread dump on query
> execution time and after query executed. Both time thread stack trace got
> different.
>
> If any other things are required then let me know I will send.
>
> Regards,
> Vishal Patel
> ________________________________
> From: Mikhail Khludnev <mk...@apache.org>
> Sent: Saturday, May 16, 2020 2:23 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
> 34 seconds? Please share the deepest thread stack. This might give a clue
> what's going on there.
>
> On Sat, May 16, 2020 at 11:46 AM vishal patel <
> vishalpatel200928@outlook.com>
> wrote:
>
> > Any one is looking my issue? Please help me.
> >
> > Sent from Outlook<http://aka.ms/weboutlook>
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Friday, May 15, 2020 3:06 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I have result of query debug for both version so It will helpful.
> >
> > Solr 6.1 query debug URL
> > https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> > Solr 8.3.1 query debug URL
> > https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
> >
> > I indexed same data in both version.
> >
> > I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> > of Solr 8.6.1. Is there any impact of score in query execution? why is
> > score=1.0 in result of Solr 8.3.0?
> >
> > Regards,
> > Vishal Patel
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Thursday, May 14, 2020 7:39 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
> >
> > I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> > when values of one field is large in query and group field is apply.
> >
> > My Solr URL :
> > https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> > My Solr config and schema :
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> <
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> >
> >
> > It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> > seconds in Solr 6.1.0.
> >
> > Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
> >
> >
> > Regards,
> > Vishal Patel
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
Thanks for reply.
I know Query field value is large. But same thing is working fine in Solr 6.1.0 and query executed within 300 milliseconds. Schema.xml and Solrconfig.xml are same. Why is it taking lots of time for execution in Solr 8.3.0?
Is there any changes in Solr 8.3.0?
Regards,
Vishal Patel
________________________________
From: Mikhail Khludnev <mk...@apache.org>
Sent: Saturday, May 16, 2020 6:55 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
It seems this thread is doing heavy work, mind the bottom line.
202.8013ms
124.8008ms
qtp153245266-156 (156)
org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.<init>(BM25Similarity.java:219)
org.apache.lucene.search.similarities.BM25Similarity.scorer(BM25Similarity.java:192)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.scorer(PerFieldSimilarityWrapper.java:47)
org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:74)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:205)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:63)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:231)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.TopFieldCollector.populateScores(TopFieldCollector.java:531)
org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.postCollect(TopGroupsFieldCommand.java:178)
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:168)
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1403)
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:387)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
It seems like it ranks groups by query score, that doubtful thing to do.
From the log. Here's how to recognize query running 25 sec "QTime=25063"
Query itself q=+msg_id:(10519539+10519540+10523575+10523576+ ... is
not what search engines are made for. They are purposed for short
query.
You may
1. leverage {!terms} query parser which might handle such long terms
list more efficiently
2. make sure you don't enable unnecessary grouping features, eg group
ranking in the stack above makes no sense for this kind of query
It's worth to revamp an overall approach in favor of query time
{!join} or index time join see {!parent}/nested docs.
On Sat, May 16, 2020 at 1:46 PM vishal patel <vi...@outlook.com>
wrote:
> Thanks for reply.
>
> I have taken a thread dump at the time of query execution. I do not know
> the thread name so send the All threads. I have also send the logs so you
> can get idea.
>
> Thread Dump All Stack Trace:
> https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
> Solr 8.3 shard 1 log:
> https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
> Solr 8.3 shard 2 log:
> https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
>
> I have some questions regarding the thread dump
> - How can I know the my thread name from thread dump? can I get from the
> log?
> - When do I take a thread dump? on query execution or after query
> execution?
>
> Note: I got a thread name from log and checked in thread dump on query
> execution time and after query executed. Both time thread stack trace got
> different.
>
> If any other things are required then let me know I will send.
>
> Regards,
> Vishal Patel
> ________________________________
> From: Mikhail Khludnev <mk...@apache.org>
> Sent: Saturday, May 16, 2020 2:23 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
> 34 seconds? Please share the deepest thread stack. This might give a clue
> what's going on there.
>
> On Sat, May 16, 2020 at 11:46 AM vishal patel <
> vishalpatel200928@outlook.com>
> wrote:
>
> > Any one is looking my issue? Please help me.
> >
> > Sent from Outlook<http://aka.ms/weboutlook>
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Friday, May 15, 2020 3:06 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I have result of query debug for both version so It will helpful.
> >
> > Solr 6.1 query debug URL
> > https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> > Solr 8.3.1 query debug URL
> > https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
> >
> > I indexed same data in both version.
> >
> > I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> > of Solr 8.6.1. Is there any impact of score in query execution? why is
> > score=1.0 in result of Solr 8.3.0?
> >
> > Regards,
> > Vishal Patel
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Thursday, May 14, 2020 7:39 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
> >
> > I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> > when values of one field is large in query and group field is apply.
> >
> > My Solr URL :
> > https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> > My Solr config and schema :
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> <
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> >
> >
> > It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> > seconds in Solr 6.1.0.
> >
> > Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
> >
> >
> > Regards,
> > Vishal Patel
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by Mikhail Khludnev <mk...@apache.org>.
It seems this thread is doing heavy work, mind the bottom line.
202.8013ms
124.8008ms
qtp153245266-156 (156)
org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer.<init>(BM25Similarity.java:219)
org.apache.lucene.search.similarities.BM25Similarity.scorer(BM25Similarity.java:192)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.scorer(PerFieldSimilarityWrapper.java:47)
org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:74)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:205)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:63)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:231)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
org.apache.lucene.search.TopFieldCollector.populateScores(TopFieldCollector.java:531)
org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.postCollect(TopGroupsFieldCommand.java:178)
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:168)
org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1403)
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:387)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
It seems like it ranks groups by query score, that doubtful thing to do.
From the log. Here's how to recognize query running 25 sec "QTime=25063"
Query itself q=+msg_id:(10519539+10519540+10523575+10523576+ ... is
not what search engines are made for. They are purposed for short
query.
You may
1. leverage {!terms} query parser which might handle such long terms
list more efficiently
2. make sure you don't enable unnecessary grouping features, eg group
ranking in the stack above makes no sense for this kind of query
It's worth to revamp an overall approach in favor of query time
{!join} or index time join see {!parent}/nested docs.
On Sat, May 16, 2020 at 1:46 PM vishal patel <vi...@outlook.com>
wrote:
> Thanks for reply.
>
> I have taken a thread dump at the time of query execution. I do not know
> the thread name so send the All threads. I have also send the logs so you
> can get idea.
>
> Thread Dump All Stack Trace:
> https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
> Solr 8.3 shard 1 log:
> https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
> Solr 8.3 shard 2 log:
> https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
>
> I have some questions regarding the thread dump
> - How can I know the my thread name from thread dump? can I get from the
> log?
> - When do I take a thread dump? on query execution or after query
> execution?
>
> Note: I got a thread name from log and checked in thread dump on query
> execution time and after query executed. Both time thread stack trace got
> different.
>
> If any other things are required then let me know I will send.
>
> Regards,
> Vishal Patel
> ________________________________
> From: Mikhail Khludnev <mk...@apache.org>
> Sent: Saturday, May 16, 2020 2:23 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
> 34 seconds? Please share the deepest thread stack. This might give a clue
> what's going on there.
>
> On Sat, May 16, 2020 at 11:46 AM vishal patel <
> vishalpatel200928@outlook.com>
> wrote:
>
> > Any one is looking my issue? Please help me.
> >
> > Sent from Outlook<http://aka.ms/weboutlook>
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Friday, May 15, 2020 3:06 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I have result of query debug for both version so It will helpful.
> >
> > Solr 6.1 query debug URL
> > https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> > Solr 8.3.1 query debug URL
> > https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
> >
> > I indexed same data in both version.
> >
> > I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> > of Solr 8.6.1. Is there any impact of score in query execution? why is
> > score=1.0 in result of Solr 8.3.0?
> >
> > Regards,
> > Vishal Patel
> > ________________________________
> > From: vishal patel <vi...@outlook.com>
> > Sent: Thursday, May 14, 2020 7:39 PM
> > To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> > Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
> >
> > I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
> >
> > I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> > when values of one field is large in query and group field is apply.
> >
> > My Solr URL :
> > https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> > My Solr config and schema :
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> <
> > https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn
> >
> >
> > It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> > seconds in Solr 6.1.0.
> >
> > Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
> >
> >
> > Regards,
> > Vishal Patel
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
Thanks for reply.
I have taken a thread dump at the time of query execution. I do not know the thread name so send the All threads. I have also send the logs so you can get idea.
Thread Dump All Stack Trace:
https://drive.google.com/file/d/1N4rVXJoaAwNvPIY2aw57gKA9mb4vRTMR/view
Solr 8.3 shard 1 log:
https://drive.google.com/file/d/1h5d_eZfQvYET7JKzbNKZwhZ_RmaX7hWf/view
Solr 8.3 shard 2 log:
https://drive.google.com/file/d/19CRflzQ7n5BZBNaaC7EFszgzKKlPfIVl/view
I have some questions regarding the thread dump
- How can I know the my thread name from thread dump? can I get from the log?
- When do I take a thread dump? on query execution or after query execution?
Note: I got a thread name from log and checked in thread dump on query execution time and after query executed. Both time thread stack trace got different.
If any other things are required then let me know I will send.
Regards,
Vishal Patel
________________________________
From: Mikhail Khludnev <mk...@apache.org>
Sent: Saturday, May 16, 2020 2:23 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
34 seconds? Please share the deepest thread stack. This might give a clue
what's going on there.
On Sat, May 16, 2020 at 11:46 AM vishal patel <vi...@outlook.com>
wrote:
> Any one is looking my issue? Please help me.
>
> Sent from Outlook<http://aka.ms/weboutlook>
> ________________________________
> From: vishal patel <vi...@outlook.com>
> Sent: Friday, May 15, 2020 3:06 PM
> To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> I have result of query debug for both version so It will helpful.
>
> Solr 6.1 query debug URL
> https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> Solr 8.3.1 query debug URL
> https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
>
> I indexed same data in both version.
>
> I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> of Solr 8.6.1. Is there any impact of score in query execution? why is
> score=1.0 in result of Solr 8.3.0?
>
> Regards,
> Vishal Patel
> ________________________________
> From: vishal patel <vi...@outlook.com>
> Sent: Thursday, May 14, 2020 7:39 PM
> To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
>
> I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> when values of one field is large in query and group field is apply.
>
> My Solr URL :
> https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> My Solr config and schema :
> https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn<
> https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn>
>
> It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> seconds in Solr 6.1.0.
>
> Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
>
>
> Regards,
> Vishal Patel
>
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by Mikhail Khludnev <mk...@apache.org>.
Can you check Thread Dump in Solr Admin while Solr 8.3 crunches query for
34 seconds? Please share the deepest thread stack. This might give a clue
what's going on there.
On Sat, May 16, 2020 at 11:46 AM vishal patel <vi...@outlook.com>
wrote:
> Any one is looking my issue? Please help me.
>
> Sent from Outlook<http://aka.ms/weboutlook>
> ________________________________
> From: vishal patel <vi...@outlook.com>
> Sent: Friday, May 15, 2020 3:06 PM
> To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> I have result of query debug for both version so It will helpful.
>
> Solr 6.1 query debug URL
> https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
> Solr 8.3.1 query debug URL
> https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
>
> I indexed same data in both version.
>
> I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result
> of Solr 8.6.1. Is there any impact of score in query execution? why is
> score=1.0 in result of Solr 8.3.0?
>
> Regards,
> Vishal Patel
> ________________________________
> From: vishal patel <vi...@outlook.com>
> Sent: Thursday, May 14, 2020 7:39 PM
> To: solr-user@lucene.apache.org <so...@lucene.apache.org>
> Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
>
> I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
>
> I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1
> when values of one field is large in query and group field is apply.
>
> My Solr URL :
> https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
> My Solr config and schema :
> https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn<
> https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn>
>
> It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5
> seconds in Solr 6.1.0.
>
> Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
>
>
> Regards,
> Vishal Patel
>
>
--
Sincerely yours
Mikhail Khludnev
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
Any one is looking my issue? Please help me.
Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Friday, May 15, 2020 3:06 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
I have result of query debug for both version so It will helpful.
Solr 6.1 query debug URL
https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
Solr 8.3.1 query debug URL
https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
I indexed same data in both version.
I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result of Solr 8.6.1. Is there any impact of score in query execution? why is score=1.0 in result of Solr 8.3.0?
Regards,
Vishal Patel
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Thursday, May 14, 2020 7:39 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1 when values of one field is large in query and group field is apply.
My Solr URL : https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
My Solr config and schema : https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn<https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn>
It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5 seconds in Solr 6.1.0.
Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
Regards,
Vishal Patel
Re: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
Posted by vishal patel <vi...@outlook.com>.
I have result of query debug for both version so It will helpful.
Solr 6.1 query debug URL
https://drive.google.com/file/d/1ixqpgAXsVLDZA-aUobJLrMOOefZX2NL1/view
Solr 8.3.1 query debug URL
https://drive.google.com/file/d/1MOKVE-iPZFuzRnDZhY9V6OsAKFT38U5r/view
I indexed same data in both version.
I found score=1.0 in result of Solr 8.3.0 and score=0.016147947 in result of Solr 8.6.1. Is there any impact of score in query execution? why is score=1.0 in result of Solr 8.3.0?
Regards,
Vishal Patel
________________________________
From: vishal patel <vi...@outlook.com>
Sent: Thursday, May 14, 2020 7:39 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Performance issue in Query execution in Solr 8.3.0 and 8.5.1
I am upgrading Solr 6.1.0 to Solr 8.3.0 or Solr 8.5.1.
I get performance issue for query execution in Solr 8.3.0 or Solr 8.5.1 when values of one field is large in query and group field is apply.
My Solr URL : https://drive.google.com/file/d/1UqFE8I6M451Z1wWAu5_C1dzqYEOGjuH2/view
My Solr config and schema : https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn<https://drive.google.com/drive/folders/1pJBxL0OOwAJSEC5uK_87ikaHEVGdDEEn>
It takes 34 seconds in Solr 8.3.0 or Solr 8.5.1. Same URL takes 1.5 seconds in Solr 6.1.0.
Is there any changes or issue related to grouping in Solr 8.3.0 or 8.5.1?
Regards,
Vishal Patel