BooleanQuery.maxClauseCount exception

[Himanshu Karki <hi...@universal-sw.com>]

Hi,

We are using Lucene.net 2.1.0.3 version for our website. With the latest index files of size around 62GB it is hitting the 'BooleanQuery.maxClauseCount' exception while doing the search sometimes and surely when search is being done by wildcard queries.

I had browsed JIRA for the reported issues and found that it is the 'memory-trade off'. My website runs perfectly fine when I replace 1024 (Default value) with  'Int.MaxValue'. But to make it more optimal,  I would prefer the below solution as mentioned in LuceneFAQ (http://wiki.apache.org/jakarta-lucene/LuceneFAQ#Why_am_I_getting_a_TooManyClauses_exception.3F) -

"Use a filter to replace the part of the query that causes the exception. For example, a RangeFilter can replace a RangeQuery on date fields and it will never throw the TooManyClauses exception -- You can even use ConstantScoreRangeQuery to execute your RangeFilter as a Query. Note that filters are slower than queries when used for the first time, so you should cache them using CachingWrapperFilter<http://lucene.apache.org/java/docs/api/org/apache/lucene/search/CachingWrapperFilter.html>. Using Filters in place of Queries generated by QueryParser can be achieved by subclassing QueryParser and overriding the appropriate function to return a ConstantScore version of your Query."

But I am unable to understand it completely. Need help to understand and implement it. Preferable would be code-snippet.


Cheers,

HIMANSHU KARKI

RE: BooleanQuery.maxClauseCount exception

["Granroth, Neal V." <ne...@thermofisher.com>]

Filters are not difficult to use, the difficult part is examining the problem searches the users are submitting and determine how a filter can be applied to improve the situation.

Here are a few code fragments from a Lucene .NET v1.9 project I developed.  A filter is being used to limit the set of "documents" that can match the user-entered query string.

     BooleanQuery bq = new BooleanQuery();

     bq.Add( Lucene.Net.QueryParsers.QueryParser.Parse( criteria,
        "_contents", new StandardAnalyzer() ), BooleanClause.Occur.SHOULD);

     Filter fx = new CachingWrapperFilter(
        new MultiRangeFilter(includeValues, excludeValues, tolerance) );

     IndexSearcher vIS = GetSearcher();

     Hits hits = vIS.Search( bq, fx );

In my situation the filter augments the types of criteria the their relationships the user can specify for a search using metadata generated and stored in the index when the "documents" were added.

The Filter itself simply sets the bits in a large BitArray indicating which documents in the index satisfy the values provided to the constructor of my MultiRangeFilter class.

Hope this helps.

Neal



-----Original Message-----
From: Himanshu Karki [mailto:himanshuk@universal-sw.com]
Sent: Tuesday, October 13, 2009 2:26 AM
To: lucene-net-dev@incubator.apache.org
Subject: BooleanQuery.maxClauseCount exception

Hi,

We are using Lucene.net 2.1.0.3 version for our website. With the latest index files of size around 62GB it is hitting the 'BooleanQuery.maxClauseCount' exception while doing the search sometimes and surely when search is being done by wildcard queries.

I had browsed JIRA for the reported issues and found that it is the 'memory-trade off'. My website runs perfectly fine when I replace 1024 (Default value) with  'Int.MaxValue'. But to make it more optimal,  I would prefer the below solution as mentioned in LuceneFAQ (http://wiki.apache.org/jakarta-lucene/LuceneFAQ#Why_am_I_getting_a_TooManyClauses_exception.3F) -

"Use a filter to replace the part of the query that causes the exception. For example, a RangeFilter can replace a RangeQuery on date fields and it will never throw the TooManyClauses exception -- You can even use ConstantScoreRangeQuery to execute your RangeFilter as a Query. Note that filters are slower than queries when used for the first time, so you should cache them using CachingWrapperFilter<http://lucene.apache.org/java/docs/api/org/apache/lucene/search/CachingWrapperFilter.html>. Using Filters in place of Queries generated by QueryParser can be achieved by subclassing QueryParser and overriding the appropriate function to return a ConstantScore version of your Query."

But I am unable to understand it completely. Need help to understand and implement it. Preferable would be code-snippet.


Cheers,

HIMANSHU KARKI