You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Giovanni Fernandez-Kincade <gf...@capitaliq.com> on 2009/10/23 00:22:59 UTC
Too Many Boolean Clauses
Hi,
I'm trying to perform a search against an integer field with a ton of OR statements for each of the unique values that I want to search for. I pasted an example at the bottom of this email. Solr fires back the following error:
org.apache.lucene.queryParser.ParseException: Cannot parse .. ': too many boolean clauses
Can you configure the maximum number of boolean clauses in lucene? Is there a better way to perform this type of search? I can't use numeric range queries because we often need to search for a huge number of IDs that are non-continuous (e.g. 4, 64, 1003, 2490, etc.). I feel like anyone that is combining SOLR with a relational database and has Solr documents referencing DB entities must have run into this issue.
Thanks,
Gio.
Field being searched:
<field name="formTypeId" type="integer" indexed="true" stored="false" multiValued="true"/>
Example query:
+companyId:32393 +formTypeId:(5 OR 6 OR 59 OR 125 OR 126 OR 135 OR 136 OR 197 OR 198 OR 209 OR 210 OR 211 OR 212 OR 213 OR 214 OR 230 OR 231 OR 232 OR 233 OR 298 OR 299 OR 377 OR 378 OR 394 OR 395 OR 402 OR 428 OR 429 OR 659 OR 670 OR 671 OR 825 OR 894 OR 3245 OR 3246 OR 3247 OR 3248 OR 3249 OR 3250 OR 3251 OR 3252 OR 3253 OR 3254 OR 3255 OR 3256 OR 3257 OR 3258 OR 3259 OR 3260 OR 3261 OR 3262 OR 3263 OR 3264 OR 3265 OR 3266 OR 3267 OR 3268 OR 3269 OR 3270 OR 3275 OR 3276 OR 3277 OR 3280 OR 3281 OR 3282 OR 3283 OR 3284 OR 3285 OR 3286 OR 3287 OR 3288 OR 3289 OR 3290 OR 3291 OR 3292 OR 3293 OR 3294 OR 3295 OR 3296 OR 3297 OR 3298 OR 3299 OR 3300 OR 3301 OR 3302 OR 3303 OR 3304 OR 3305 OR 3306 OR 3307 OR 3308 OR 3309 OR 3310 OR 3311 OR 3312 OR 3313 OR 3314 OR 3315 OR 3316 OR 3317 OR 3318 OR 3319 OR 3320 OR 3321 OR 3322 OR 3323 OR 3324 OR 3325 OR 3892 OR 3893 OR 3894 OR 3895 OR 3896 OR 3897 OR 3900 OR 3901 OR 3903 OR 3904 OR 3905 OR 3906 OR 3907 OR 3908 OR 3909 OR 3910 OR 3911 OR 3912 OR 3913 OR 5012 OR 5013 OR 5014 OR 5015 OR 5016 OR 5019 OR 5020 OR 5021 OR 5022 OR 5023 OR 5024 OR 5025 OR 5026 OR 5027 OR 5028 OR 5029 OR 5030 OR 6069 OR 6070 OR 6084 OR 6085 OR ...
Re: Too Many Boolean Clauses
Posted by Mark Miller <ma...@gmail.com>.
Hmm - yeah - coming at that a bit from the Lucene perspective - you
wouldn't want it cached. For a ton of clauses like that, a filter can
outperform a booleanquery on a large index in certain cases (an uncached
filter). And doesn't suffer from having to mess with the max clause
setting. Thats part of why the Multi-Term queries now use a constant
score mode (essentially a filter) when matching a lot of terms. They use
a constant score boolean query mode when matching fewer terms.
Giovanni Fernandez-Kincade wrote:
> Is a filter really appropriate in this case? These types of condition would never get re-used - they would be unique to each search performed by users in our application.
>
> Is there any performance benefit to the filter aside from caching (which wouldn't do much for us anyway)?
>
>
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Thursday, October 22, 2009 6:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Too Many Boolean Clauses
>
> Giovanni Fernandez-Kincade wrote:
>
>> Hi,
>> I'm trying to perform a search against an integer field with a ton of OR statements for each of the unique values that I want to search for. I pasted an example at the bottom of this email. Solr fires back the following error:
>> org.apache.lucene.queryParser.ParseException: Cannot parse .. ': too many boolean clauses
>> Can you configure the maximum number of boolean clauses in lucene? Is there a better way to perform this type of search? I can't use numeric range queries because we often need to search for a huge number of IDs that are non-continuous (e.g. 4, 64, 1003, 2490, etc.). I feel like anyone that is combining SOLR with a relational database and has Solr documents referencing DB entities must have run into this issue.
>>
>> Thanks,
>> Gio.
>>
>> Field being searched:
>> <field name="formTypeId" type="integer" indexed="true" stored="false" multiValued="true"/>
>>
>> Example query:
>> +companyId:32393 +formTypeId:(5 OR 6 OR 59 OR 125 OR 126 OR 135 OR 136 OR 197 OR 198 OR 209 OR 210 OR 211 OR 212 OR 213 OR 214 OR 230 OR 231 OR 232 OR 233 OR 298 OR 299 OR 377 OR 378 OR 394 OR 395 OR 402 OR 428 OR 429 OR 659 OR 670 OR 671 OR 825 OR 894 OR 3245 OR 3246 OR 3247 OR 3248 OR 3249 OR 3250 OR 3251 OR 3252 OR 3253 OR 3254 OR 3255 OR 3256 OR 3257 OR 3258 OR 3259 OR 3260 OR 3261 OR 3262 OR 3263 OR 3264 OR 3265 OR 3266 OR 3267 OR 3268 OR 3269 OR 3270 OR 3275 OR 3276 OR 3277 OR 3280 OR 3281 OR 3282 OR 3283 OR 3284 OR 3285 OR 3286 OR 3287 OR 3288 OR 3289 OR 3290 OR 3291 OR 3292 OR 3293 OR 3294 OR 3295 OR 3296 OR 3297 OR 3298 OR 3299 OR 3300 OR 3301 OR 3302 OR 3303 OR 3304 OR 3305 OR 3306 OR 3307 OR 3308 OR 3309 OR 3310 OR 3311 OR 3312 OR 3313 OR 3314 OR 3315 OR 3316 OR 3317 OR 3318 OR 3319 OR 3320 OR 3321 OR 3322 OR 3323 OR 3324 OR 3325 OR 3892 OR 3893 OR 3894 OR 3895 OR 3896 OR 3897 OR 3900 OR 3901 OR 3903 OR 3904 OR 3905 OR 3906 OR 3907 OR 3908 OR 3909 OR 3910 OR 3911 OR 3912 OR 3913 OR 5012 OR 5013 OR 5014 OR 5015 OR 5016 OR 5019 OR 5020 OR 5021 OR 5022 OR 5023 OR 5024 OR 5025 OR 5026 OR 5027 OR 5028 OR 5029 OR 5030 OR 6069 OR 6070 OR 6084 OR 6085 OR ...
>>
>>
>>
> Yuck. Might want to try making a filter instead. Or something else.
>
> You can raise the maxBooleanClause limit in solrconfig.xml. You can find
> an example in the example config.
>
>
--
- Mark
http://www.lucidimagination.com
RE: Too Many Boolean Clauses
Posted by Giovanni Fernandez-Kincade <gf...@capitaliq.com>.
Is a filter really appropriate in this case? These types of condition would never get re-used - they would be unique to each search performed by users in our application.
Is there any performance benefit to the filter aside from caching (which wouldn't do much for us anyway)?
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Thursday, October 22, 2009 6:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Too Many Boolean Clauses
Giovanni Fernandez-Kincade wrote:
> Hi,
> I'm trying to perform a search against an integer field with a ton of OR statements for each of the unique values that I want to search for. I pasted an example at the bottom of this email. Solr fires back the following error:
> org.apache.lucene.queryParser.ParseException: Cannot parse .. ': too many boolean clauses
> Can you configure the maximum number of boolean clauses in lucene? Is there a better way to perform this type of search? I can't use numeric range queries because we often need to search for a huge number of IDs that are non-continuous (e.g. 4, 64, 1003, 2490, etc.). I feel like anyone that is combining SOLR with a relational database and has Solr documents referencing DB entities must have run into this issue.
>
> Thanks,
> Gio.
>
> Field being searched:
> <field name="formTypeId" type="integer" indexed="true" stored="false" multiValued="true"/>
>
> Example query:
> +companyId:32393 +formTypeId:(5 OR 6 OR 59 OR 125 OR 126 OR 135 OR 136 OR 197 OR 198 OR 209 OR 210 OR 211 OR 212 OR 213 OR 214 OR 230 OR 231 OR 232 OR 233 OR 298 OR 299 OR 377 OR 378 OR 394 OR 395 OR 402 OR 428 OR 429 OR 659 OR 670 OR 671 OR 825 OR 894 OR 3245 OR 3246 OR 3247 OR 3248 OR 3249 OR 3250 OR 3251 OR 3252 OR 3253 OR 3254 OR 3255 OR 3256 OR 3257 OR 3258 OR 3259 OR 3260 OR 3261 OR 3262 OR 3263 OR 3264 OR 3265 OR 3266 OR 3267 OR 3268 OR 3269 OR 3270 OR 3275 OR 3276 OR 3277 OR 3280 OR 3281 OR 3282 OR 3283 OR 3284 OR 3285 OR 3286 OR 3287 OR 3288 OR 3289 OR 3290 OR 3291 OR 3292 OR 3293 OR 3294 OR 3295 OR 3296 OR 3297 OR 3298 OR 3299 OR 3300 OR 3301 OR 3302 OR 3303 OR 3304 OR 3305 OR 3306 OR 3307 OR 3308 OR 3309 OR 3310 OR 3311 OR 3312 OR 3313 OR 3314 OR 3315 OR 3316 OR 3317 OR 3318 OR 3319 OR 3320 OR 3321 OR 3322 OR 3323 OR 3324 OR 3325 OR 3892 OR 3893 OR 3894 OR 3895 OR 3896 OR 3897 OR 3900 OR 3901 OR 3903 OR 3904 OR 3905 OR 3906 OR 3907 OR 3908 OR 3909 OR 3910 OR 3911 OR 3912 OR 3913 OR 5012 OR 5013 OR 5014 OR 5015 OR 5016 OR 5019 OR 5020 OR 5021 OR 5022 OR 5023 OR 5024 OR 5025 OR 5026 OR 5027 OR 5028 OR 5029 OR 5030 OR 6069 OR 6070 OR 6084 OR 6085 OR ...
>
>
Yuck. Might want to try making a filter instead. Or something else.
You can raise the maxBooleanClause limit in solrconfig.xml. You can find
an example in the example config.
--
- Mark
http://www.lucidimagination.com
Re: Too Many Boolean Clauses
Posted by Mark Miller <ma...@gmail.com>.
Giovanni Fernandez-Kincade wrote:
> Hi,
> I'm trying to perform a search against an integer field with a ton of OR statements for each of the unique values that I want to search for. I pasted an example at the bottom of this email. Solr fires back the following error:
> org.apache.lucene.queryParser.ParseException: Cannot parse .. ': too many boolean clauses
> Can you configure the maximum number of boolean clauses in lucene? Is there a better way to perform this type of search? I can't use numeric range queries because we often need to search for a huge number of IDs that are non-continuous (e.g. 4, 64, 1003, 2490, etc.). I feel like anyone that is combining SOLR with a relational database and has Solr documents referencing DB entities must have run into this issue.
>
> Thanks,
> Gio.
>
> Field being searched:
> <field name="formTypeId" type="integer" indexed="true" stored="false" multiValued="true"/>
>
> Example query:
> +companyId:32393 +formTypeId:(5 OR 6 OR 59 OR 125 OR 126 OR 135 OR 136 OR 197 OR 198 OR 209 OR 210 OR 211 OR 212 OR 213 OR 214 OR 230 OR 231 OR 232 OR 233 OR 298 OR 299 OR 377 OR 378 OR 394 OR 395 OR 402 OR 428 OR 429 OR 659 OR 670 OR 671 OR 825 OR 894 OR 3245 OR 3246 OR 3247 OR 3248 OR 3249 OR 3250 OR 3251 OR 3252 OR 3253 OR 3254 OR 3255 OR 3256 OR 3257 OR 3258 OR 3259 OR 3260 OR 3261 OR 3262 OR 3263 OR 3264 OR 3265 OR 3266 OR 3267 OR 3268 OR 3269 OR 3270 OR 3275 OR 3276 OR 3277 OR 3280 OR 3281 OR 3282 OR 3283 OR 3284 OR 3285 OR 3286 OR 3287 OR 3288 OR 3289 OR 3290 OR 3291 OR 3292 OR 3293 OR 3294 OR 3295 OR 3296 OR 3297 OR 3298 OR 3299 OR 3300 OR 3301 OR 3302 OR 3303 OR 3304 OR 3305 OR 3306 OR 3307 OR 3308 OR 3309 OR 3310 OR 3311 OR 3312 OR 3313 OR 3314 OR 3315 OR 3316 OR 3317 OR 3318 OR 3319 OR 3320 OR 3321 OR 3322 OR 3323 OR 3324 OR 3325 OR 3892 OR 3893 OR 3894 OR 3895 OR 3896 OR 3897 OR 3900 OR 3901 OR 3903 OR 3904 OR 3905 OR 3906 OR 3907 OR 3908 OR 3909 OR 3910 OR 3911 OR 3912 OR 3913 OR 5012 OR 5013 OR 5014 OR 5015 OR 5016 OR 5019 OR 5020 OR 5021 OR 5022 OR 5023 OR 5024 OR 5025 OR 5026 OR 5027 OR 5028 OR 5029 OR 5030 OR 6069 OR 6070 OR 6084 OR 6085 OR ...
>
>
Yuck. Might want to try making a filter instead. Or something else.
You can raise the maxBooleanClause limit in solrconfig.xml. You can find
an example in the example config.
--
- Mark
http://www.lucidimagination.com