You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Phillip Farber <pf...@umich.edu> on 2008/04/03 17:03:22 UTC
limit on number of values in a filter query?
I use a filter query (fq) parameter in my requests to limit the select
response to a subset of all document ids. I'm getting a Solr exception
when the number of values in the fq approaches 1000. I get the
following response from Solr:
_header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
... <<several 100's of ids>> ... +851832976+8536687}{}
_buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
... <<several 100's of ids>> ... +851832976+8536687}{}
2008-04-03 10:42:54.005::WARN: handle failed
java.io.IOException: FULL
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
The id field is currently a string. Would it help to make it a long
int? Is there some internal limit on the number of values in a fq or is
there a configuration parameter that will allow me to increase this number?
Thanks,
Phil
Re: limit on number of values in a filter query?
Posted by Phillip Farber <pf...@umich.edu>.
It is a POST so it's not a jetty limit.
The mysterious "4096" numbers in the string that begins
_header=[6518349,32231686,m=4,g=4096,p=4096,c=4096] ... and
"java.io.IOException: FULL" in the server traceback made me think that
the size of a static buffer is being exceeded. Since the exception is
I/O could it be an I/O buffer limitation?
The fq values represent the document ids in a collection. Is there a
workaround to allow longer fq lists? I guess one way is to limit
collection size but this is not desirable for application reasons.
Phil
Yonik Seeley wrote:
> 1000 filters, wow!
>
> Perhaps you hit a jetty limit on the size of a GET request or
> something? Perhaps try POST?
>
> -Yonik
>
> On Thu, Apr 3, 2008 at 11:03 AM, Phillip Farber <pf...@umich.edu> wrote:
>> I use a filter query (fq) parameter in my requests to limit the select
>> response to a subset of all document ids. I'm getting a Solr exception when
>> the number of values in the fq approaches 1000. I get the following
>> response from Solr:
>>
>>
>> _header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>> ... <<several 100's of ids>> ... +851832976+8536687}{}
>>
>> _buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>> ... <<several 100's of ids>> ... +851832976+8536687}{}
>> 2008-04-03 10:42:54.005::WARN: handle failed
>> java.io.IOException: FULL
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>> at
>> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>>
>>
>> The id field is currently a string. Would it help to make it a long int?
>> Is there some internal limit on the number of values in a fq or is there a
>> configuration parameter that will allow me to increase this number?
>>
>> Thanks,
>>
>> Phil
>>
Re: limit on number of values in a filter query?
Posted by Yonik Seeley <yo...@apache.org>.
1000 filters, wow!
Perhaps you hit a jetty limit on the size of a GET request or
something? Perhaps try POST?
-Yonik
On Thu, Apr 3, 2008 at 11:03 AM, Phillip Farber <pf...@umich.edu> wrote:
> I use a filter query (fq) parameter in my requests to limit the select
> response to a subset of all document ids. I'm getting a Solr exception when
> the number of values in the fq approaches 1000. I get the following
> response from Solr:
>
>
> _header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
> ... <<several 100's of ids>> ... +851832976+8536687}{}
>
> _buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
> ... <<several 100's of ids>> ... +851832976+8536687}{}
> 2008-04-03 10:42:54.005::WARN: handle failed
> java.io.IOException: FULL
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>
> The id field is currently a string. Would it help to make it a long int?
> Is there some internal limit on the number of values in a fq or is there a
> configuration parameter that will allow me to increase this number?
>
> Thanks,
>
> Phil
>