You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by yliu <yl...@mathworks.com> on 2015/04/28 21:58:09 UTC
How to improve the performance of query with expand query
Hi,
I am using Solr to do some complex querying. Some queries require me to
return one main document and some expanded documents at the same time. When
I run the query without extracting any expanded document, the performance is
good. But once I added the expanded conditions, the performance becomes
really bad. What is the best way to get a document and a few related
documents (child documents) back at the same time? Is there anything I can
config or change in schema.xml file and solrconfig.xml file to help improve
the performance? Thanks,
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by yliu <yl...@mathworks.com>.
I will try it out with Solr 5. Thanks for your help,
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203030.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by yliu <yl...@mathworks.com>.
One more question. The 70516592 includes everything in the core, not only
those have the same _root_. My expand.field limit the results to the same
root. There are only 382 documents with the same _root_. Would that have
ruled out most of the other documents? Maybe I have misunderstood your
question. When you say how many documents in index, do you mean the whole
core or do you mean just documents related to the same parent? Thanks,
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203031.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by Joel Bernstein <jo...@gmail.com>.
Ok, this explains the slowness. The expand.q returns 380 documents but it's
actually postfiltering the entire 70 million+ result set in the case of
expand.q=*:*. The problem is the postfilter only approach that Expand uses
in Solr 4.x.
Expand was originally designed to use the main query as the expand.q, which
typically produces smaller result sets. It was also designed mainly to be
used with Collapse, where it's main job was to expand the collapsed result
set, which would have been limited by the main query.
Solr 5, will be much faster for the scenario that you are using.
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Apr 29, 2015 at 10:58 AM, yliu <yl...@mathworks.com> wrote:
> There are totally 70516592 documents in the index and 380 is returned from
> the expand.q query. How do I include licenseNumber in expand.q query?
> None
> of the expand documents have that field.
>
> Thanks,
>
> yliu
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203023.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to improve the performance of query with expand query
Posted by yliu <yl...@mathworks.com>.
There are totally 70516592 documents in the index and 380 is returned from
the expand.q query. How do I include licenseNumber in expand.q query? None
of the expand documents have that field.
Thanks,
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203023.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by Joel Bernstein <jo...@gmail.com>.
Is it possible to use the licenseNumber is the expand.q? In Solr 4*, Expand
brings back all the results in the expand.q and maps them to groups. So
large result sets generated by expand.q can be slow.
In Solr 5, Expand limits the results coming back from expand.q to only
include records that have an expand.field that is included in the main
result page. This would be much faster in the scenario you are describing.
How many document are in the index? How many documents are returned by this
query:
expand.q=contentType:ActivationBean
OR contentType:EntitlementReleaseBean OR contentType:EntitlementProductBean
OR (contentType:EntitlementPersonBean AND
webProfileId:5449738)
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Apr 29, 2015 at 10:26 AM, yliu <yl...@mathworks.com> wrote:
> What I have here is the complete request. I am not using any "collapse" in
> my query.
>
> Before the queries I listed in my previous post, I have only:
> http://..../solr-slave/entitlement/select/?
>
> Thanks.
>
> yliu
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203010.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to improve the performance of query with expand query
Posted by yliu <yl...@mathworks.com>.
What I have here is the complete request. I am not using any "collapse" in
my query.
Before the queries I listed in my previous post, I have only:
http://..../solr-slave/entitlement/select/?
Thanks.
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4203010.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by Joel Bernstein <jo...@gmail.com>.
Can you paste in the entire http request? Wondering if you're using both
collapse and expand. Both of these have an effect on performance.
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Apr 29, 2015 at 8:48 AM, yliu <yl...@mathworks.com> wrote:
> Hi, Joel,
>
> Here are some details:
>
> 1) I am using solr version 4.10.4.
>
> When my request is: q=contentType:EntitlementBean AND
> licenseNumber:40281071.
>
> It returns in 414ms. And the response size is 4728 bytes. One document
> returned.
>
> When my request is: q=contentType:EntitlementBean AND
>
> licenseNumber:40281071&expand=true&expand.field=_root_&expand.q=*:*&expand.rows=20000
>
> It returns in 1253ms. And the response size is 220865 bytes. One document
> returned. 382 expanded documents returned.
>
> When my request is: q=contentType:EntitlementBean AND
>
> licenseNumber:40281071&expand=true&expand.field=_root_&expand.q=contentType:ActivationBean
> OR contentType:EntitlementReleaseBean OR contentType:EntitlementProductBean
> OR (contentType:EntitlementPersonBean AND
> webProfileId:5449738)&expand.rows=20000
>
> It returns in 11541ms. And the response size is 220319 bytes. One
> document
> returned. 380 expanded documents returned.
>
> Please let me know if you need more information. Thanks,
>
> yliu
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4202991.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to improve the performance of query with expand query
Posted by yliu <yl...@mathworks.com>.
Hi, Joel,
Here are some details:
1) I am using solr version 4.10.4.
When my request is: q=contentType:EntitlementBean AND
licenseNumber:40281071.
It returns in 414ms. And the response size is 4728 bytes. One document
returned.
When my request is: q=contentType:EntitlementBean AND
licenseNumber:40281071&expand=true&expand.field=_root_&expand.q=*:*&expand.rows=20000
It returns in 1253ms. And the response size is 220865 bytes. One document
returned. 382 expanded documents returned.
When my request is: q=contentType:EntitlementBean AND
licenseNumber:40281071&expand=true&expand.field=_root_&expand.q=contentType:ActivationBean
OR contentType:EntitlementReleaseBean OR contentType:EntitlementProductBean
OR (contentType:EntitlementPersonBean AND
webProfileId:5449738)&expand.rows=20000
It returns in 11541ms. And the response size is 220319 bytes. One document
returned. 380 expanded documents returned.
Please let me know if you need more information. Thanks,
yliu
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895p4202991.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to improve the performance of query with expand query
Posted by Joel Bernstein <jo...@gmail.com>.
Could you provide a few more details?
1) Version of Lucene/Solr
2) A sample slow query
3) Number of unique values in the collapse field
4) Number of search results before the collapse
5) Number of results fetched in the page
6) Performance numbers for the query
Joel Bernstein
http://joelsolr.blogspot.com/
On Tue, Apr 28, 2015 at 3:58 PM, yliu <yl...@mathworks.com> wrote:
> Hi,
>
> I am using Solr to do some complex querying. Some queries require me to
> return one main document and some expanded documents at the same time.
> When
> I run the query without extracting any expanded document, the performance
> is
> good. But once I added the expanded conditions, the performance becomes
> really bad. What is the best way to get a document and a few related
> documents (child documents) back at the same time? Is there anything I can
> config or change in schema.xml file and solrconfig.xml file to help improve
> the performance? Thanks,
>
> yliu
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-improve-the-performance-of-query-with-expand-query-tp4202895.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>