You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Venkateswarlu Bommineni <bv...@gmail.com> on 2019/05/30 20:20:57 UTC
Solr query with long query
Hello Team,
I have got a requirement to send many strings (~1500) in the filter query
param to the solr.
Can you please provide any suggestions/precautions we need to take care in
this particular scenario.
Thanks,
Venkat.
Re: Solr query with long query
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
You can use POST instead of GET.
But you may also want to see if you can refactor those 1500 strings somehow.
If you don't use it already, maybe Terms query parser could be useful:
https://lucene.apache.org/solr/guide/7_7/other-parsers.html#terms-query-parser
Also, if at least some of those numbers don't change, maybe you can
store them Solr side with default request parameters or even using
Request Param API with useParams:
https://lucene.apache.org/solr/guide/7_7/request-parameters-api.html
Regards,
Alex.
On Thu, 30 May 2019 at 16:21, Venkateswarlu Bommineni <bv...@gmail.com> wrote:
>
> Hello Team,
>
> I have got a requirement to send many strings (~1500) in the filter query
> param to the solr.
>
> Can you please provide any suggestions/precautions we need to take care in
> this particular scenario.
>
> Thanks,
> Venkat.
Re: Solr query with long query
Posted by Tim Casey <tc...@gmail.com>.
Venkat,
There is another way to do this. If you have a category of "thing" you are
attempting to filter over, then you create a query and tag the documents
with this category. So, create a 'categories' field and append 'thing' to
the field updating the field if need be. (Be wary of over generation if
one of the categories turns out to be 'thin').
Then in the filter query you can query over a category, or simply require a
category:thing to be in the query.
tim
On Thu, May 30, 2019 at 3:33 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 5/30/2019 4:13 PM, Venkateswarlu Bommineni wrote:
> > Thank you guys for quick response.
> >
> > I was able to query solr by sending 1500 products using solrJ with http
> > post method.
> >
> > But I had to change maxBooleanClauses to 4096 from default 1024.
> >
> > But I wanted to check with you guys that, will there be any performance
> > issues by changing maxBooleanClauses to 4096.
> > Please help me here.
>
> Lucene sets the limit to 1024 basically to act as a short-circuit to
> protect the system against bugs in user code causing the query engine to
> explode by sending thousands of clauses when that wasn't what was intended.
>
> Using a large number of clauses *can* cause performance issues. But if
> you need to use them, you need to use them, and in that case you'll need
> to increase the limit, and just take the performance hit.
>
> The terms query parser will generally run faster than a large number of
> boolean OR clauses, and return the same results. But if your query is
> not just OR (also known as SHOULD) clauses, you probably won't be able
> to use the terms parser.
>
>
> https://lucene.apache.org/solr/guide/7_7/other-parsers.html#terms-query-parser
>
> Thanks,
> Shawn
>
Re: Solr query with long query
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/30/2019 4:13 PM, Venkateswarlu Bommineni wrote:
> Thank you guys for quick response.
>
> I was able to query solr by sending 1500 products using solrJ with http
> post method.
>
> But I had to change maxBooleanClauses to 4096 from default 1024.
>
> But I wanted to check with you guys that, will there be any performance
> issues by changing maxBooleanClauses to 4096.
> Please help me here.
Lucene sets the limit to 1024 basically to act as a short-circuit to
protect the system against bugs in user code causing the query engine to
explode by sending thousands of clauses when that wasn't what was intended.
Using a large number of clauses *can* cause performance issues. But if
you need to use them, you need to use them, and in that case you'll need
to increase the limit, and just take the performance hit.
The terms query parser will generally run faster than a large number of
boolean OR clauses, and return the same results. But if your query is
not just OR (also known as SHOULD) clauses, you probably won't be able
to use the terms parser.
https://lucene.apache.org/solr/guide/7_7/other-parsers.html#terms-query-parser
Thanks,
Shawn
Re: Solr query with long query
Posted by Venkateswarlu Bommineni <bv...@gmail.com>.
Thank you guys for quick response.
I was able to query solr by sending 1500 products using solrJ with http
post method.
But I had to change maxBooleanClauses to 4096 from default 1024.
But I wanted to check with you guys that, will there be any performance
issues by changing maxBooleanClauses to 4096.
Please help me here.
Thanks,
Venkat.
On Thu, May 30, 2019 at 2:46 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 5/30/2019 2:20 PM, Venkateswarlu Bommineni wrote:
> > I have got a requirement to send many strings (~1500) in the filter query
> > param to the solr.
> >
> > Can you please provide any suggestions/precautions we need to take care
> in
> > this particular scenario.
>
> You'll probably want to send that as a POST, where the URL parameters
> are in the HTTP body, not the URL. With a standard GET request, there
> is no body. Like most web servers, the Jetty included with Solr has an
> HTTP header size limit of 8192 bytes, which affects how large a URL you
> can use. The default size limit on the request body for POST requests
> is two megabytes.
>
> Depending on which query parser it ends up using, you might also need to
> increase maxBooleanClauses in all your solrconfig.xml files. If this is
> a situation where it's all "OR" clauses, it's recommended to use the
> terms query parser, which has a minimum Solr version requirement of
> 4.10. That query parser doesn't care about maxBooleanClauses. Some
> later versions of Solr will automatically convert lots of OR clauses
> into the type of query used by the terms parser, but I do not know which
> versions do that.
>
> Thanks,
> Shawn
>
Re: Solr query with long query
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/30/2019 2:20 PM, Venkateswarlu Bommineni wrote:
> I have got a requirement to send many strings (~1500) in the filter query
> param to the solr.
>
> Can you please provide any suggestions/precautions we need to take care in
> this particular scenario.
You'll probably want to send that as a POST, where the URL parameters
are in the HTTP body, not the URL. With a standard GET request, there
is no body. Like most web servers, the Jetty included with Solr has an
HTTP header size limit of 8192 bytes, which affects how large a URL you
can use. The default size limit on the request body for POST requests
is two megabytes.
Depending on which query parser it ends up using, you might also need to
increase maxBooleanClauses in all your solrconfig.xml files. If this is
a situation where it's all "OR" clauses, it's recommended to use the
terms query parser, which has a minimum Solr version requirement of
4.10. That query parser doesn't care about maxBooleanClauses. Some
later versions of Solr will automatically convert lots of OR clauses
into the type of query used by the terms parser, but I do not know which
versions do that.
Thanks,
Shawn
Re: Solr query with long query
Posted by Erick Erickson <er...@gmail.com>.
I’m talking about the filterCache. You said you were using this in a “filter query”, which I assume is an “fq” clause, which is automatically cached in the filterCache.
It would help a lot if you told us two things:
1> what the clause looks like. You say 1,500 strings. How are you assmebling them?
2> what the field definition is…
Best,
Erick
> On May 31, 2019, at 12:53 AM, Venkateswarlu Bommineni <bv...@gmail.com> wrote:
>
> Tim/Eric,
>
> @Tim : We do have category field in existing solr configuration. So the
> existing functionality is we will query based on category and get the
> results from solr and display them on PCAT.
>
> But as per new requirement , we need to invoke third party service to fetch
> the personalized products and send them to Solr get the results. that is
> the reason i was tying to send those products in solr query.
>
> @Eric : we are using Solr 7.2.1 version. I will check on solr cache.
>
> Thanks,
> Venkat.
>
> On Thu, May 30, 2019 at 7:53 PM Erick Erickson <er...@gmail.com>
> wrote:
>
>> What version of Solr? Recent versions automatically use terms query parser
>> for large, simple or clauses. Do look into using it anyway. And I'd set
>> cache=false because I doubt you'll ever get a cache hit...
>>
>> On Thu, May 30, 2019, 16:21 Venkateswarlu Bommineni <bv...@gmail.com>
>> wrote:
>>
>>> Hello Team,
>>>
>>> I have got a requirement to send many strings (~1500) in the filter query
>>> param to the solr.
>>>
>>> Can you please provide any suggestions/precautions we need to take care
>> in
>>> this particular scenario.
>>>
>>> Thanks,
>>> Venkat.
>>>
>>
Re: Solr query with long query
Posted by Venkateswarlu Bommineni <bv...@gmail.com>.
Tim/Eric,
@Tim : We do have category field in existing solr configuration. So the
existing functionality is we will query based on category and get the
results from solr and display them on PCAT.
But as per new requirement , we need to invoke third party service to fetch
the personalized products and send them to Solr get the results. that is
the reason i was tying to send those products in solr query.
@Eric : we are using Solr 7.2.1 version. I will check on solr cache.
Thanks,
Venkat.
On Thu, May 30, 2019 at 7:53 PM Erick Erickson <er...@gmail.com>
wrote:
> What version of Solr? Recent versions automatically use terms query parser
> for large, simple or clauses. Do look into using it anyway. And I'd set
> cache=false because I doubt you'll ever get a cache hit...
>
> On Thu, May 30, 2019, 16:21 Venkateswarlu Bommineni <bv...@gmail.com>
> wrote:
>
> > Hello Team,
> >
> > I have got a requirement to send many strings (~1500) in the filter query
> > param to the solr.
> >
> > Can you please provide any suggestions/precautions we need to take care
> in
> > this particular scenario.
> >
> > Thanks,
> > Venkat.
> >
>
Re: Solr query with long query
Posted by Erick Erickson <er...@gmail.com>.
What version of Solr? Recent versions automatically use terms query parser
for large, simple or clauses. Do look into using it anyway. And I'd set
cache=false because I doubt you'll ever get a cache hit...
On Thu, May 30, 2019, 16:21 Venkateswarlu Bommineni <bv...@gmail.com>
wrote:
> Hello Team,
>
> I have got a requirement to send many strings (~1500) in the filter query
> param to the solr.
>
> Can you please provide any suggestions/precautions we need to take care in
> this particular scenario.
>
> Thanks,
> Venkat.
>