You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by mitu2009 <mu...@gmail.com> on 2009/10/08 01:42:43 UTC
Help needed figuring out reason for maxClauseCount is set to 1024
error
Hi,
I've two sets of search indexes. TestIndex (used in our test environment)
and ProdIndex(used in PRODUCTION environment). Lucene search query:
+date:[20090410184806 TO 20091007184806] works fine for test index but gives
this error message for Prod index.
"maxClauseCount is set to 1024"
If I execute following line just before executing search query, then I do
not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
searcher.Search(myQuery, collector);
Am I missing something here?Why am not getting this error in test index?The
schema for two indexes are same.They only differ wrt to number of
records/data.PROD index has got higher number of records(around 1300) than
those in test one (around 950).
Thanks for reading.
--
View this message in context: http://www.nabble.com/Help-needed-figuring-out-reason-for-maxClauseCount-is-set-to-1024-error-tp25796342p25796342.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Help needed figuring out reason for maxClauseCount is set to 1024
error
Posted by Adriano Crestani <ad...@gmail.com>.
Hi,
Can you provide to us the exception stack trace?
Thanks,
Adriano Crestani
On Wed, Oct 7, 2009 at 7:42 PM, mitu2009 <mu...@gmail.com> wrote:
>
> Hi,
>
> I've two sets of search indexes. TestIndex (used in our test environment)
> and ProdIndex(used in PRODUCTION environment). Lucene search query:
> +date:[20090410184806 TO 20091007184806] works fine for test index but
> gives
> this error message for Prod index.
>
> "maxClauseCount is set to 1024"
>
> If I execute following line just before executing search query, then I do
> not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
> searcher.Search(myQuery, collector);
>
> Am I missing something here?Why am not getting this error in test index?The
> schema for two indexes are same.They only differ wrt to number of
> records/data.PROD index has got higher number of records(around 1300) than
> those in test one (around 950).
>
> Thanks for reading.
>
> --
> View this message in context:
> http://www.nabble.com/Help-needed-figuring-out-reason-for-maxClauseCount-is-set-to-1024-error-tp25796342p25796342.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Help needed figuring out reason for maxClauseCount is set to 1024 error
Posted by Uwe Schindler <uw...@thetaphi.de>.
The precision depends on how you convert you datetime to an integer value.
For NumericRangeQuery the precision doesn't really matter. Only index gets
bigger. If you want day resolution just divide Date.getTime() by e.g.
86400000L (which is milliseconds). You have to do the same conversation on
both indexing and searching, but this should be clear.
NumericRangeQuery is good for all fields, which cardinality is >100. But it
doesn't hurt for lower precisions.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Jake Mannix [mailto:jake.mannix@gmail.com]
> Sent: Thursday, October 08, 2009 7:24 AM
> To: java-user@lucene.apache.org
> Subject: Re: Help needed figuring out reason for maxClauseCount is set to
> 1024 error
>
> When such precision is needed, this is a great idea. When it's far more
> than
> overkill (like when only days are necessary), is there anything to gain by
> doing this?
>
> -jake
>
>
> On Wed, Oct 7, 2009 at 10:17 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > I would propose to use NumericRangeQuery and NumericField supplied by
> > Lucene
> > 2.9. This has no such limitations. You can index your dates as numeric
> > value
> > (e.g. Date.getTime()) and query downto the milliseconds.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Jake Mannix [mailto:jake.mannix@gmail.com]
> > > Sent: Thursday, October 08, 2009 2:35 AM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Help needed figuring out reason for maxClauseCount is set
> to
> > > 1024 error
> > >
> > > On Wed, Oct 7, 2009 at 4:42 PM, mitu2009 <mu...@gmail.com>
> wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > I've two sets of search indexes. TestIndex (used in our test
> > > environment)
> > > > and ProdIndex(used in PRODUCTION environment). Lucene search query:
> > > > +date:[20090410184806 TO 20091007184806] works fine for test index
> but
> > > > gives
> > > > this error message for Prod index.
> > > >
> > >
> > > Oooh, this date range is going to kill your performance. You need to
> use
> > > less
> > > precision unless you really *need* accuracy down to the second.
> > >
> > >
> > > >
> > > > "maxClauseCount is set to 1024"
> > > >
> > > > If I execute following line just before executing search query, then
> I
> > > do
> > > > not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
> > > > searcher.Search(myQuery, collector);
> > > >
> > > > Am I missing something here?Why am not getting this error in test
> > > index?The
> > > > schema for two indexes are same.They only differ wrt to number of
> > > > records/data.PROD index has got higher number of records(around
> 1300)
> > > than
> > > > those in test one (around 950).
> > > >
> > >
> > > The reason why it is different in PROD is that your production index
> has
> > > more than
> > > a 1024 different values for your date field, while in test, your index
> is
> > > small enough
> > > so that there are less than 1024 values, so when lucene turns the
> range
> > > query into
> > > a BooleanQuery, it blows up in prod, but not in test, due to the
> number
> > of
> > > terms
> > > it must be rewritten into.
> > >
> > > Make sense?
> > >
> > > -jake
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Help needed figuring out reason for maxClauseCount is set to 1024
error
Posted by Jake Mannix <ja...@gmail.com>.
When such precision is needed, this is a great idea. When it's far more
than
overkill (like when only days are necessary), is there anything to gain by
doing this?
-jake
On Wed, Oct 7, 2009 at 10:17 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> I would propose to use NumericRangeQuery and NumericField supplied by
> Lucene
> 2.9. This has no such limitations. You can index your dates as numeric
> value
> (e.g. Date.getTime()) and query downto the milliseconds.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Jake Mannix [mailto:jake.mannix@gmail.com]
> > Sent: Thursday, October 08, 2009 2:35 AM
> > To: java-user@lucene.apache.org
> > Subject: Re: Help needed figuring out reason for maxClauseCount is set to
> > 1024 error
> >
> > On Wed, Oct 7, 2009 at 4:42 PM, mitu2009 <mu...@gmail.com> wrote:
> >
> > >
> > > Hi,
> > >
> > > I've two sets of search indexes. TestIndex (used in our test
> > environment)
> > > and ProdIndex(used in PRODUCTION environment). Lucene search query:
> > > +date:[20090410184806 TO 20091007184806] works fine for test index but
> > > gives
> > > this error message for Prod index.
> > >
> >
> > Oooh, this date range is going to kill your performance. You need to use
> > less
> > precision unless you really *need* accuracy down to the second.
> >
> >
> > >
> > > "maxClauseCount is set to 1024"
> > >
> > > If I execute following line just before executing search query, then I
> > do
> > > not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
> > > searcher.Search(myQuery, collector);
> > >
> > > Am I missing something here?Why am not getting this error in test
> > index?The
> > > schema for two indexes are same.They only differ wrt to number of
> > > records/data.PROD index has got higher number of records(around 1300)
> > than
> > > those in test one (around 950).
> > >
> >
> > The reason why it is different in PROD is that your production index has
> > more than
> > a 1024 different values for your date field, while in test, your index is
> > small enough
> > so that there are less than 1024 values, so when lucene turns the range
> > query into
> > a BooleanQuery, it blows up in prod, but not in test, due to the number
> of
> > terms
> > it must be rewritten into.
> >
> > Make sense?
> >
> > -jake
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Help needed figuring out reason for maxClauseCount is set to 1024 error
Posted by Uwe Schindler <uw...@thetaphi.de>.
I would propose to use NumericRangeQuery and NumericField supplied by Lucene
2.9. This has no such limitations. You can index your dates as numeric value
(e.g. Date.getTime()) and query downto the milliseconds.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Jake Mannix [mailto:jake.mannix@gmail.com]
> Sent: Thursday, October 08, 2009 2:35 AM
> To: java-user@lucene.apache.org
> Subject: Re: Help needed figuring out reason for maxClauseCount is set to
> 1024 error
>
> On Wed, Oct 7, 2009 at 4:42 PM, mitu2009 <mu...@gmail.com> wrote:
>
> >
> > Hi,
> >
> > I've two sets of search indexes. TestIndex (used in our test
> environment)
> > and ProdIndex(used in PRODUCTION environment). Lucene search query:
> > +date:[20090410184806 TO 20091007184806] works fine for test index but
> > gives
> > this error message for Prod index.
> >
>
> Oooh, this date range is going to kill your performance. You need to use
> less
> precision unless you really *need* accuracy down to the second.
>
>
> >
> > "maxClauseCount is set to 1024"
> >
> > If I execute following line just before executing search query, then I
> do
> > not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
> > searcher.Search(myQuery, collector);
> >
> > Am I missing something here?Why am not getting this error in test
> index?The
> > schema for two indexes are same.They only differ wrt to number of
> > records/data.PROD index has got higher number of records(around 1300)
> than
> > those in test one (around 950).
> >
>
> The reason why it is different in PROD is that your production index has
> more than
> a 1024 different values for your date field, while in test, your index is
> small enough
> so that there are less than 1024 values, so when lucene turns the range
> query into
> a BooleanQuery, it blows up in prod, but not in test, due to the number of
> terms
> it must be rewritten into.
>
> Make sense?
>
> -jake
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Help needed figuring out reason for maxClauseCount is set to 1024
error
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, Oct 7, 2009 at 4:42 PM, mitu2009 <mu...@gmail.com> wrote:
>
> Hi,
>
> I've two sets of search indexes. TestIndex (used in our test environment)
> and ProdIndex(used in PRODUCTION environment). Lucene search query:
> +date:[20090410184806 TO 20091007184806] works fine for test index but
> gives
> this error message for Prod index.
>
Oooh, this date range is going to kill your performance. You need to use
less
precision unless you really *need* accuracy down to the second.
>
> "maxClauseCount is set to 1024"
>
> If I execute following line just before executing search query, then I do
> not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue);
> searcher.Search(myQuery, collector);
>
> Am I missing something here?Why am not getting this error in test index?The
> schema for two indexes are same.They only differ wrt to number of
> records/data.PROD index has got higher number of records(around 1300) than
> those in test one (around 950).
>
The reason why it is different in PROD is that your production index has
more than
a 1024 different values for your date field, while in test, your index is
small enough
so that there are less than 1024 values, so when lucene turns the range
query into
a BooleanQuery, it blows up in prod, but not in test, due to the number of
terms
it must be rewritten into.
Make sense?
-jake