You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by dipanjan_pramanick <di...@infosys.com> on 2009/07/20 10:07:43 UTC
Help needed with Solr
Hi,
We have scenario where we need to send more than 1024 ids in the Solr url as "OR" condition.
I have changed the value of maxBooleanClauses in solrconfig.xml, to 2048, but it is failing after handling 1024 "OR" conditions.
Solr is throwing "SEVERE: org.apache.solr.common.SolrException: Bad Request" whenever I am sending more than 1024 OR conditions. Is there any way I can change this value on Solr configuration.
Thanks,
Dipanjan
Re: Help needed with Solr
Posted by Avlesh Singh <av...@gmail.com>.
If yours is a JAVA stack of application, I would recommend moving to SolrJ.
It is a client API which lets you talk to Solr. Know more about it here -
http://wiki.apache.org/solr/Solrj
Clients API's for other languages can be found here -
http://wiki.apache.org/solr/#head-ab1768efa59b26cbd30f1acd03b633f1d110ed47
Cheers
Avlesh
On Mon, Jul 20, 2009 at 3:44 PM, dipanjan_pramanick <
dipanjan_pramanick@infosys.com> wrote:
> Hi Shalin,
>
> We just found that there is no limit on Solr side about the maximum boolean
> condition. We have set the <maxBooleanClauses>2048</maxBooleanClauses> and
> we are able to send about 1574 OR conditions.
> Over that limit, we are getting "HTTP/1.1 400 Bad Request".
>
> You are correct, it's not a Solr issue, its due to HTTP GET is not being
> able to send such a large request.
>
> But now the question is, Solr only accepts request in url form not as
> request parameter or request object. That's the main issue. Hence we need to
> send the query in url form only.
> Can you please suggest, if you have tried the similar thing instead of
> passing as URL, instead passing as object or request entity using POST.
>
>
>
> Thanks,
> Dipanjan
>
>
> ________________________________
> From: Shalin Shekhar Mangar <sh...@gmail.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Mon, 20 Jul 2009 14:54:04 +0530
> To: <so...@lucene.apache.org>
> Subject: Re: Help needed with Solr <maxBooleanClauses>
>
> On Mon, Jul 20, 2009 at 2:12 PM, dipanjan_pramanick <
> dipanjan_pramanick@infosys.com> wrote:
>
> >
> > Its true that there is a design flaw, because of what we need to support
> a
> > huge list of OR conditions through Solr.
> > But still I would like to know if there is any other configuration other
> > than the one in solrConfig.xml, through which we can pass more than 1024
> OR
> > conditions.
> >
> > <maxBooleanClauses>1024<maxBooleanClauses>
> >
> >
> > Regarding HTTP Post, in Solr 1.3, it is only accepts request in url form
> > not as request parameter or request object. That's another issue. Hence
> we
> > need to send the query in url form only.
> >
> >
> Changing the value of maxBooleanClauses in solrconfig.xml is sufficient.
> The
> problem here is that you may be exceeding the maximum allowed size of an
> HTTP GET request (is that 2KB?). You must use POST request to send such a
> huge query string. Again, it will help if you can post the complete stack
> trace of the error.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
> **************** CAUTION - Disclaimer *****************
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended
> solely
> for the use of the addressee(s). If you are not the intended recipient,
> please
> notify the sender by e-mail and delete the original message. Further, you
> are not
> to copy, disclose, or distribute this e-mail or its contents to any other
> person and
> any such actions are unlawful. This e-mail may contain viruses. Infosys has
> taken
> every reasonable precaution to minimize this risk, but is not liable for
> any damage
> you may sustain as a result of any virus in this e-mail. You should carry
> out your
> own virus checks before opening the e-mail or attachment. Infosys reserves
> the
> right to monitor and review the content of all messages sent to or from
> this e-mail
> address. Messages sent to or from this e-mail address may be stored on the
> Infosys e-mail system.
> ***INFOSYS******** End of Disclaimer ********INFOSYS***
>
Re: Help needed with Solr
Posted by Chris Hostetter <ho...@fucit.org>.
: You are correct, it's not a Solr issue, its due to HTTP GET is not being
: able to send such a large request.
:
: But now the question is, Solr only accepts request in url form not as
: request parameter or request object. That's the main issue. Hence we
: need to send the query in url form only.
There's a couple of of missleading statements and confusion above that i
want to clarify for anyone reading...
1) there is no limit to how long a GET string can be in HTTP, there are
only limits imposed by server/client implementatations, if your servlet
container and your client support infinitely long URLs, Solr will work
fine.
2) you can absolutely POST queries to Solr if you set the content type
appropriately.
furthermore: if you are sending queries that large, you should seriously
consider refactoring your problem and thinking about why the queries are
so large, and wether you can put some of the functionality that generates
those large queries into a Solr plugin so you don't need to send so much
data over the wire.
-Hoss
Re: Help needed with Solr
Posted by dipanjan_pramanick <di...@infosys.com>.
Hi Shalin,
We just found that there is no limit on Solr side about the maximum boolean condition. We have set the <maxBooleanClauses>2048</maxBooleanClauses> and we are able to send about 1574 OR conditions.
Over that limit, we are getting "HTTP/1.1 400 Bad Request".
You are correct, it's not a Solr issue, its due to HTTP GET is not being able to send such a large request.
But now the question is, Solr only accepts request in url form not as request parameter or request object. That's the main issue. Hence we need to send the query in url form only.
Can you please suggest, if you have tried the similar thing instead of passing as URL, instead passing as object or request entity using POST.
Thanks,
Dipanjan
________________________________
From: Shalin Shekhar Mangar <sh...@gmail.com>
Reply-To: <so...@lucene.apache.org>
Date: Mon, 20 Jul 2009 14:54:04 +0530
To: <so...@lucene.apache.org>
Subject: Re: Help needed with Solr <maxBooleanClauses>
On Mon, Jul 20, 2009 at 2:12 PM, dipanjan_pramanick <
dipanjan_pramanick@infosys.com> wrote:
>
> Its true that there is a design flaw, because of what we need to support a
> huge list of OR conditions through Solr.
> But still I would like to know if there is any other configuration other
> than the one in solrConfig.xml, through which we can pass more than 1024 OR
> conditions.
>
> <maxBooleanClauses>1024<maxBooleanClauses>
>
>
> Regarding HTTP Post, in Solr 1.3, it is only accepts request in url form
> not as request parameter or request object. That's another issue. Hence we
> need to send the query in url form only.
>
>
Changing the value of maxBooleanClauses in solrconfig.xml is sufficient. The
problem here is that you may be exceeding the maximum allowed size of an
HTTP GET request (is that 2KB?). You must use POST request to send such a
huge query string. Again, it will help if you can post the complete stack
trace of the error.
--
Regards,
Shalin Shekhar Mangar.
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
Re: Help needed with Solr
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Jul 20, 2009 at 2:12 PM, dipanjan_pramanick <
dipanjan_pramanick@infosys.com> wrote:
>
> Its true that there is a design flaw, because of what we need to support a
> huge list of OR conditions through Solr.
> But still I would like to know if there is any other configuration other
> than the one in solrConfig.xml, through which we can pass more than 1024 OR
> conditions.
>
> <maxBooleanClauses>1024<maxBooleanClauses>
>
>
> Regarding HTTP Post, in Solr 1.3, it is only accepts request in url form
> not as request parameter or request object. That's another issue. Hence we
> need to send the query in url form only.
>
>
Changing the value of maxBooleanClauses in solrconfig.xml is sufficient. The
problem here is that you may be exceeding the maximum allowed size of an
HTTP GET request (is that 2KB?). You must use POST request to send such a
huge query string. Again, it will help if you can post the complete stack
trace of the error.
--
Regards,
Shalin Shekhar Mangar.
Re: Help needed with Solr
Posted by dipanjan_pramanick <di...@infosys.com>.
Hi Shalin,
Thanks for your time to respond to this issue.
Its true that there is a design flaw, because of what we need to support a huge list of OR conditions through Solr.
But still I would like to know if there is any other configuration other than the one in solrConfig.xml, through which we can pass more than 1024 OR conditions.
<maxBooleanClauses>1024<maxBooleanClauses>
Regarding HTTP Post, in Solr 1.3, it is only accepts request in url form not as request parameter or request object. That's another issue. Hence we need to send the query in url form only.
Thanks,
Dipanjan
________________________________
From: Shalin Shekhar Mangar <sh...@gmail.com>
Reply-To: <so...@lucene.apache.org>
Date: Mon, 20 Jul 2009 13:58:55 +0530
To: <so...@lucene.apache.org>
Subject: Re: Help needed with Solr <maxBooleanClauses>
On Mon, Jul 20, 2009 at 1:37 PM, dipanjan_pramanick <
dipanjan_pramanick@infosys.com> wrote:
> Hi,
> We have scenario where we need to send more than 1024 ids in the Solr url
> as "OR" condition.
> I have changed the value of maxBooleanClauses in solrconfig.xml, to 2048,
> but it is failing after handling 1024 "OR" conditions.
> Solr is throwing "SEVERE: org.apache.solr.common.SolrException: Bad
> Request" whenever I am sending more than 1024 OR conditions. Is there any
> way I can change this value on Solr configuration.
The maxBooleanClauses is there as a safe guard against extremely slow
queries. If you can tell us about the exact problem you are solving, we may
be able to suggest an alternative approach? Creating such huge boolean
clauses may be a bad design choice.
As for the exception you are seeing, it seems to me that you may be
exceeding the size of a GET request. Using an HTTP POST request may work.
--
Regards,
Shalin Shekhar Mangar.
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
Re: Help needed with Solr
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Jul 20, 2009 at 1:37 PM, dipanjan_pramanick <
dipanjan_pramanick@infosys.com> wrote:
> Hi,
> We have scenario where we need to send more than 1024 ids in the Solr url
> as "OR" condition.
> I have changed the value of maxBooleanClauses in solrconfig.xml, to 2048,
> but it is failing after handling 1024 "OR" conditions.
> Solr is throwing "SEVERE: org.apache.solr.common.SolrException: Bad
> Request" whenever I am sending more than 1024 OR conditions. Is there any
> way I can change this value on Solr configuration.
The maxBooleanClauses is there as a safe guard against extremely slow
queries. If you can tell us about the exact problem you are solving, we may
be able to suggest an alternative approach? Creating such huge boolean
clauses may be a bad design choice.
As for the exception you are seeing, it seems to me that you may be
exceeding the size of a GET request. Using an HTTP POST request may work.
--
Regards,
Shalin Shekhar Mangar.