You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by michael dürr <du...@gmail.com> on 2022/12/01 12:27:19 UTC
Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Hi,
today we updated solr to version 9.1 (lucene version 9.3)
Since then we noticed plenty of TooManyNestedClauses in the logs. Our
setting for maxClauseCount is 1024
I played around a lot and could trace it down to this:
* I built an index from scratch with two fields (id is unique key) and
luceneMatchVersion 9.3:
<field name="id" type="string_dv" indexed="true" stored="true"
multiValued="false" required="true"/>
<field name="createdById" type="p_long_dv" indexed="true" stored="false"
multiValued="false" />
<fieldType name="string_dv" class="solr.StrField" sortMissingLast="true"
omitNorms="true" docValues="true" />
fieldType name="p_long_dv" class="solr.LongPointField" docValues="true"
omitNorms="true" />
As expected this works (the dots(...) represent the complete set of numbers
up to 1024):
curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ... 1024)
And this fails:
curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ... 1025)
But when I use the other field (categoryId) this fails:
curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1 2 3
... 1024)
It works until 512 and starts failing from 513 clauses
No difference when doing it like this:
curl -XGET http://localhost:8983/solr/myindex/select?q=+(categoryId:1
categoryId:2 ... categoryId:1024)
Am I misunderstanding the limit maxClauseCount?
I'm pretty sure that we did not have any issues with this before.
Thanks,
Michael
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by michael dürr <du...@gmail.com>.
Hi Hoss,
This is a really helpful explanation!
Even though I already shifted to the usage of the {!terms} query for such
large boolean clause queries, it feels a lot better to know how and why
things behave differently compared to the 8x solr version.
Thanks!
Michael
On Tue, Dec 6, 2022 at 7:32 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : I'm happy to provide some details as I still do not really understand the
> : difference to the situation before.
>
> The main difference is coming from the changes introduced in LUCENE-8811
> (Lucene 9.0) which sought to ensure that the "global" maxClauseCount would
> be honored no matter what kind of nested structure the query might
> involve.
>
> You're situation is an interesting case that i had never considered, more
> detais below...
>
> : * I upgraded from 8.11.1 to 9.1. I observed the behavior for a completely
> : rebuild index (solr version 9.1 / lucene version 9.3)
>
> thank you for clarifing. This confirms that changes introduced
> in LUCENE-8811 (and related solr issues) are relavant to the change in
> behavior you are seeing (if you had said you upgraded from Solr 9 we'd be
> having a different conversation)
>
> : * maxBooleanClauses is only configured in solrconfig.xml (1024) but not
> in
> : solr.xml.
>
> FYI: If you don't configure in solr.xml, then the (Lucene) default
> IndexSearcher.getMaxClauseCount() is left as is (and that is also 1024)
>
> : * Sorry for the confusion about the field definition. As you already
> : assumed correctly: 'categoryId' is also a 'p_long_dv'
>
> Meaning that it has both points nad docvalues configured, which it turns
> out is significant to why it behaves differently from a string field.
>
>
> : * Stacktrace for String field ("id"). For better readability I replaced
> the
> : original query by "1 2 ... 1025":
>
> Snipping down to the key lines of code from the root cause...
>
> : Caused by: org.apache.lucene.search.IndexSearcher$TooManyClauses:
> : maxClauseCount is set to 1024
> : at
> : org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:116)
> : at
> : org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:130)
> : at
> :
> org.apache.solr.parser.SolrQueryParserBase.rawToNormal(SolrQueryParserBase.java:1065)
>
> ...so in this case, as the query parser is building up a boolean query (of
> many strings), it is hitting the limit because the (top level) boolean
> query is being asked to add one more item then
> IndexSearcher.getMaxClauseCount() == 1024
>
>
> : * Stacktrace for Point field ("categoryId") with 1 2 ... 513:
>
> Again, snipping down to just the key lines of code. (Note also the
> difference in the exception message: "too many nested clauses") ..
>
> : org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query
> contains
> : too many nested clauses; maxClauseCount is set to 1024
> : at
> :
> org.apache.lucene.search.IndexSearcher$3.visitLeaf(IndexSearcher.java:801)
> : at
> :
> org.apache.lucene.document.SortedNumericDocValuesRangeQuery.visit(SortedNumericDocValuesRangeQuery.java:73)
> : at
> :
> org.apache.lucene.search.IndexOrDocValuesQuery.visit(IndexOrDocValuesQuery.java:121)
> : at
> : org.apache.lucene.search.BooleanQuery.visit(BooleanQuery.java:575)
> : at
> : org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:769)
>
> ...here the exception is happening during the actual search -- meaning the
> query parser had no problem building up the BooleanQuery of 512 clauses
>
> But what matters is that each of those 512 clauses is no longer a simple
> exact term query (or a simple exact point query, or a simple exact
> docvalue query) ... because this fieldType is configured to support both
> points and docvalues, those 512 clauses are IndexOrDocValuesQuery queries
> -- which each contain 2 sub-clauses
>
> (the purpose of this class is to provide teh most efficient impl based on
> where/how this clause is used, which can depend on term stats, other
> clauses in the parent query, etc...)
>
> So to sumarize:
>
> 1) the reason you're seeing this behavior in 9x but didnt' in 8x is
> because 9x added more checks of the safety valve
>
> 2) the reason you're seeing the 1024 limit hit for some (but not all)
> fields, even with with less then 1024 "original user query clauses" is
> because for some (but not all) field types, 1 original query clause can
> become N internal clauses.
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Chris Hostetter <ho...@fucit.org>.
: I'm happy to provide some details as I still do not really understand the
: difference to the situation before.
The main difference is coming from the changes introduced in LUCENE-8811
(Lucene 9.0) which sought to ensure that the "global" maxClauseCount would
be honored no matter what kind of nested structure the query might
involve.
You're situation is an interesting case that i had never considered, more
detais below...
: * I upgraded from 8.11.1 to 9.1. I observed the behavior for a completely
: rebuild index (solr version 9.1 / lucene version 9.3)
thank you for clarifing. This confirms that changes introduced
in LUCENE-8811 (and related solr issues) are relavant to the change in
behavior you are seeing (if you had said you upgraded from Solr 9 we'd be
having a different conversation)
: * maxBooleanClauses is only configured in solrconfig.xml (1024) but not in
: solr.xml.
FYI: If you don't configure in solr.xml, then the (Lucene) default
IndexSearcher.getMaxClauseCount() is left as is (and that is also 1024)
: * Sorry for the confusion about the field definition. As you already
: assumed correctly: 'categoryId' is also a 'p_long_dv'
Meaning that it has both points nad docvalues configured, which it turns
out is significant to why it behaves differently from a string field.
: * Stacktrace for String field ("id"). For better readability I replaced the
: original query by "1 2 ... 1025":
Snipping down to the key lines of code from the root cause...
: Caused by: org.apache.lucene.search.IndexSearcher$TooManyClauses:
: maxClauseCount is set to 1024
: at
: org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:116)
: at
: org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:130)
: at
: org.apache.solr.parser.SolrQueryParserBase.rawToNormal(SolrQueryParserBase.java:1065)
...so in this case, as the query parser is building up a boolean query (of
many strings), it is hitting the limit because the (top level) boolean
query is being asked to add one more item then
IndexSearcher.getMaxClauseCount() == 1024
: * Stacktrace for Point field ("categoryId") with 1 2 ... 513:
Again, snipping down to just the key lines of code. (Note also the
difference in the exception message: "too many nested clauses") ..
: org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains
: too many nested clauses; maxClauseCount is set to 1024
: at
: org.apache.lucene.search.IndexSearcher$3.visitLeaf(IndexSearcher.java:801)
: at
: org.apache.lucene.document.SortedNumericDocValuesRangeQuery.visit(SortedNumericDocValuesRangeQuery.java:73)
: at
: org.apache.lucene.search.IndexOrDocValuesQuery.visit(IndexOrDocValuesQuery.java:121)
: at
: org.apache.lucene.search.BooleanQuery.visit(BooleanQuery.java:575)
: at
: org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:769)
...here the exception is happening during the actual search -- meaning the
query parser had no problem building up the BooleanQuery of 512 clauses
But what matters is that each of those 512 clauses is no longer a simple
exact term query (or a simple exact point query, or a simple exact
docvalue query) ... because this fieldType is configured to support both
points and docvalues, those 512 clauses are IndexOrDocValuesQuery queries
-- which each contain 2 sub-clauses
(the purpose of this class is to provide teh most efficient impl based on
where/how this clause is used, which can depend on term stats, other
clauses in the parent query, etc...)
So to sumarize:
1) the reason you're seeing this behavior in 9x but didnt' in 8x is
because 9x added more checks of the safety valve
2) the reason you're seeing the 1024 limit hit for some (but not all)
fields, even with with less then 1024 "original user query clauses" is
because for some (but not all) field types, 1 original query clause can
become N internal clauses.
-Hoss
http://www.lucidworks.com/
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by michael dürr <du...@gmail.com>.
Hi Hoss,
I'm happy to provide some details as I still do not really understand the
difference to the situation before.
In case something like categoryId:[1 TO 1] also gets converted to some
boolean term, then it's clear to me.
Otherwise I do not understand why half the boolean clauses (512 + 1)
already cause that exception.
Here the details you asked for:
* I upgraded from 8.11.1 to 9.1. I observed the behavior for a completely
rebuild index (solr version 9.1 / lucene version 9.3)
* maxBooleanClauses is only configured in solrconfig.xml (1024) but not in
solr.xml.
* Sorry for the confusion about the field definition. As you already
assumed correctly: 'categoryId' is also a 'p_long_dv'
* Stacktrace for String field ("id"). For better readability I replaced the
original query by "1 2 ... 1025":
2022-12-06 07:08:46.625 ERROR (qtp1530880511-27) [ portal]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
org.apache.solr.search.SyntaxError: Cannot parse ' +id:(1 2 ... 1025 )':
too many boolean clauses
org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError:
Cannot parse ' +id:(1 2 ... 1025 )': too many boolean clauses
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:217)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:384)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2865)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:218)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
~[jetty-security-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
~[jetty-rewrite-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.solr.search.SyntaxError: Cannot parse ' +id:(1 2 ...
1025 )': too many boolean clauses
at
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:279)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:51)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.search.QParser.getQuery(QParser.java:188)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
... 45 more
Caused by: org.apache.lucene.search.IndexSearcher$TooManyClauses:
maxClauseCount is set to 1024
at
org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:116)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:130)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.solr.parser.SolrQueryParserBase.rawToNormal(SolrQueryParserBase.java:1065)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.parser.SolrQueryParserBase.getBooleanQuery(SolrQueryParserBase.java:750)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:250)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:296)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:171)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:141)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:274)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:51)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.search.QParser.getQuery(QParser.java:188)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
... 45 more
* Stacktrace for Point field ("categoryId") with 1 2 ... 513:
2022-12-06 07:12:48.619 ERROR (qtp1530880511-27) [ portal]
o.a.s.s.HttpSolrCall
org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains
too many nested clauses; maxClauseCount is set to 1024
org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains
too many nested clauses; maxClauseCount is set to 1024
at
org.apache.lucene.search.IndexSearcher$3.visitLeaf(IndexSearcher.java:801)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.document.SortedNumericDocValuesRangeQuery.visit(SortedNumericDocValuesRangeQuery.java:73)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.IndexOrDocValuesQuery.visit(IndexOrDocValuesQuery.java:121)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.BooleanQuery.visit(BooleanQuery.java:575)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:769)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:775)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:513)
~[lucene-core-9.3.0.jar:9.3.0 d25cebcef7a80369f4dfb9285ca7360a810b75dc -
ivera - 2022-07-25 12:30:23]
at
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:266)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1856)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1673)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:711)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1637)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:421)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2865)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:218)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
~[solr-core-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
~[jetty-security-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
~[jetty-rewrite-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Hope that helps!
Thanks,
Michael
On Mon, Dec 5, 2022 at 7:56 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : today we updated solr to version 9.1 (lucene version 9.3)
>
> Which version did you upgrade from?
>
> : Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> : setting for maxClauseCount is 1024
>
> Exactly where/how are you setting that? There are 2 settings related to
> this...
>
>
> https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html#global-maxbooleanclauses
>
>
> https://solr.apache.org/guide/solr/latest/configuration-guide/caches-warming.html#maxbooleanclauses-element
>
> : <field name="id" type="string_dv" indexed="true" stored="true"
> : multiValued="false" required="true"/>
> : <field name="createdById" type="p_long_dv" indexed="true" stored="false"
> : multiValued="false" />
>
> FYI: You've shown us the definition of 'createdById' but your example
> queries use 'categoryId'
>
> I'm going to assume for now that 'categoryId' is also a 'p_long_dv'
> field...
>
> : But when I use the other field (categoryId) this fails:
> :
> : curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1 2
> 3
> : ... 1024)
> :
> : It works until 512 and starts failing from 513 clauses
>
> that certainly smells like it might indicate a diff between the solr.xml
> maxBooleanClauses setting and the solrconfig.xml maxBooleanClauses --
> because one is enforced ath teh QueryParser level, and one is enforced at
> the query re-write level ... but off the top of my head i can't think of
> why your Point would trigger the query parser check but your String field
> wouldn't.
>
> In addition to the previous questions i asked about, I would really be
> interested to see the stack traces of both exceptions: from querying your
> string field with 1025 clauses, and your point field with 513 clauses.
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Chris Hostetter <ho...@fucit.org>.
: today we updated solr to version 9.1 (lucene version 9.3)
Which version did you upgrade from?
: Since then we noticed plenty of TooManyNestedClauses in the logs. Our
: setting for maxClauseCount is 1024
Exactly where/how are you setting that? There are 2 settings related to
this...
https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html#global-maxbooleanclauses
https://solr.apache.org/guide/solr/latest/configuration-guide/caches-warming.html#maxbooleanclauses-element
: <field name="id" type="string_dv" indexed="true" stored="true"
: multiValued="false" required="true"/>
: <field name="createdById" type="p_long_dv" indexed="true" stored="false"
: multiValued="false" />
FYI: You've shown us the definition of 'createdById' but your example
queries use 'categoryId'
I'm going to assume for now that 'categoryId' is also a 'p_long_dv'
field...
: But when I use the other field (categoryId) this fails:
:
: curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1 2 3
: ... 1024)
:
: It works until 512 and starts failing from 513 clauses
that certainly smells like it might indicate a diff between the solr.xml
maxBooleanClauses setting and the solrconfig.xml maxBooleanClauses --
because one is enforced ath teh QueryParser level, and one is enforced at
the query re-write level ... but off the top of my head i can't think of
why your Point would trigger the query parser check but your String field
wouldn't.
In addition to the previous questions i asked about, I would really be
interested to see the stack traces of both exceptions: from querying your
string field with 1025 clauses, and your point field with 513 clauses.
-Hoss
http://www.lucidworks.com/
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by michael dürr <du...@gmail.com>.
Hi Jan,
thanks! That helped :-)
On Fri, Dec 2, 2022 at 8:47 AM Jan Høydahl <ja...@cominvent.com> wrote:
> A plain q=id:(a b c) is parsed into a boolean query with three SHOULD
> clauses, i.e. OR. Try to add &debugQuery=true to a request and see how it
> gets parsed. Then if the limit is 1024 you'll get errors above.
>
> Jan
>
> > 2. des. 2022 kl. 07:43 skrev michael dürr <du...@gmail.com>:
> >
> > Thanks to all of you for your advice on using the terms query! I wasn't
> > aware of this syntax until now.
> >
> > Anyways it would be good to know whether I hit a bug or not.
> > Are my example queries probably rewritten to something that has more
> > boolean clauses?
> > If so, why doesn't that apply to the query for the unique key field?
> >
> > Maybe someone can give some insights here?
> >
> > Thanks,
> > Michael
> >
> > On Thu, Dec 1, 2022 at 7:45 PM Kevin Risden <kr...@apache.org> wrote:
> >
> >>
> >>
> https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
> >>
> >> The "!{terms ..." syntax is short for a query parser. Its a terms query
> >> parser and as Jan said its way more efficient than boolean clauses for a
> >> list of terms.
> >> Kevin Risden
> >>
> >>
> >>
> >> On Thu, Dec 1, 2022 at 1:04 PM Thomas Heigl <th...@umschalt.com>
> wrote:
> >>
> >>> Hi Jan,
> >>>
> >>> We ran into the same issue. Terms queries sound like the ideal solution
> >> for
> >>> our use case, but I couldn't find any documentation on the {!terms}
> >> syntax.
> >>> Is there anything in the official docs?
> >>>
> >>> Best,
> >>>
> >>> Thomas
> >>>
> >>> On Thu, Dec 1, 2022 at 2:09 PM Jan Høydahl <ja...@cominvent.com>
> >> wrote:
> >>>
> >>>> Have you tried using Terms Query? It is much more efficient than many
> >>>> boolean should clauses
> >>>>
> >>>> ?q={!terms f=id}1 2 3 4...1025
> >>>>
> >>>> Jan
> >>>>
> >>>>> 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> today we updated solr to version 9.1 (lucene version 9.3)
> >>>>> Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> >>>>> setting for maxClauseCount is 1024
> >>>>> I played around a lot and could trace it down to this:
> >>>>>
> >>>>> * I built an index from scratch with two fields (id is unique key)
> >> and
> >>>>> luceneMatchVersion 9.3:
> >>>>>
> >>>>> <field name="id" type="string_dv" indexed="true" stored="true"
> >>>>> multiValued="false" required="true"/>
> >>>>> <field name="createdById" type="p_long_dv" indexed="true"
> >>> stored="false"
> >>>>> multiValued="false" />
> >>>>>
> >>>>> <fieldType name="string_dv" class="solr.StrField"
> >>> sortMissingLast="true"
> >>>>> omitNorms="true" docValues="true" />
> >>>>> fieldType name="p_long_dv" class="solr.LongPointField"
> >> docValues="true"
> >>>>> omitNorms="true" />
> >>>>>
> >>>>> As expected this works (the dots(...) represent the complete set of
> >>>> numbers
> >>>>> up to 1024):
> >>>>>
> >>>>> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
> >> ...
> >>>> 1024)
> >>>>>
> >>>>> And this fails:
> >>>>>
> >>>>> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
> >> ...
> >>>> 1025)
> >>>>>
> >>>>> But when I use the other field (categoryId) this fails:
> >>>>>
> >>>>> curl -XGET
> >> http://localhost:8983/solr/myindex/select?q=+categoryId:(1
> >>> 2
> >>>> 3
> >>>>> ... 1024)
> >>>>>
> >>>>> It works until 512 and starts failing from 513 clauses
> >>>>>
> >>>>> No difference when doing it like this:
> >>>>>
> >>>>> curl -XGET
> >> http://localhost:8983/solr/myindex/select?q=+(categoryId:1
> >>>>> categoryId:2 ... categoryId:1024)
> >>>>>
> >>>>> Am I misunderstanding the limit maxClauseCount?
> >>>>>
> >>>>> I'm pretty sure that we did not have any issues with this before.
> >>>>>
> >>>>> Thanks,
> >>>>> Michael
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Jan Høydahl <ja...@cominvent.com>.
A plain q=id:(a b c) is parsed into a boolean query with three SHOULD clauses, i.e. OR. Try to add &debugQuery=true to a request and see how it gets parsed. Then if the limit is 1024 you'll get errors above.
Jan
> 2. des. 2022 kl. 07:43 skrev michael dürr <du...@gmail.com>:
>
> Thanks to all of you for your advice on using the terms query! I wasn't
> aware of this syntax until now.
>
> Anyways it would be good to know whether I hit a bug or not.
> Are my example queries probably rewritten to something that has more
> boolean clauses?
> If so, why doesn't that apply to the query for the unique key field?
>
> Maybe someone can give some insights here?
>
> Thanks,
> Michael
>
> On Thu, Dec 1, 2022 at 7:45 PM Kevin Risden <kr...@apache.org> wrote:
>
>>
>> https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
>>
>> The "!{terms ..." syntax is short for a query parser. Its a terms query
>> parser and as Jan said its way more efficient than boolean clauses for a
>> list of terms.
>> Kevin Risden
>>
>>
>>
>> On Thu, Dec 1, 2022 at 1:04 PM Thomas Heigl <th...@umschalt.com> wrote:
>>
>>> Hi Jan,
>>>
>>> We ran into the same issue. Terms queries sound like the ideal solution
>> for
>>> our use case, but I couldn't find any documentation on the {!terms}
>> syntax.
>>> Is there anything in the official docs?
>>>
>>> Best,
>>>
>>> Thomas
>>>
>>> On Thu, Dec 1, 2022 at 2:09 PM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>>
>>>> Have you tried using Terms Query? It is much more efficient than many
>>>> boolean should clauses
>>>>
>>>> ?q={!terms f=id}1 2 3 4...1025
>>>>
>>>> Jan
>>>>
>>>>> 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> today we updated solr to version 9.1 (lucene version 9.3)
>>>>> Since then we noticed plenty of TooManyNestedClauses in the logs. Our
>>>>> setting for maxClauseCount is 1024
>>>>> I played around a lot and could trace it down to this:
>>>>>
>>>>> * I built an index from scratch with two fields (id is unique key)
>> and
>>>>> luceneMatchVersion 9.3:
>>>>>
>>>>> <field name="id" type="string_dv" indexed="true" stored="true"
>>>>> multiValued="false" required="true"/>
>>>>> <field name="createdById" type="p_long_dv" indexed="true"
>>> stored="false"
>>>>> multiValued="false" />
>>>>>
>>>>> <fieldType name="string_dv" class="solr.StrField"
>>> sortMissingLast="true"
>>>>> omitNorms="true" docValues="true" />
>>>>> fieldType name="p_long_dv" class="solr.LongPointField"
>> docValues="true"
>>>>> omitNorms="true" />
>>>>>
>>>>> As expected this works (the dots(...) represent the complete set of
>>>> numbers
>>>>> up to 1024):
>>>>>
>>>>> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
>> ...
>>>> 1024)
>>>>>
>>>>> And this fails:
>>>>>
>>>>> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
>> ...
>>>> 1025)
>>>>>
>>>>> But when I use the other field (categoryId) this fails:
>>>>>
>>>>> curl -XGET
>> http://localhost:8983/solr/myindex/select?q=+categoryId:(1
>>> 2
>>>> 3
>>>>> ... 1024)
>>>>>
>>>>> It works until 512 and starts failing from 513 clauses
>>>>>
>>>>> No difference when doing it like this:
>>>>>
>>>>> curl -XGET
>> http://localhost:8983/solr/myindex/select?q=+(categoryId:1
>>>>> categoryId:2 ... categoryId:1024)
>>>>>
>>>>> Am I misunderstanding the limit maxClauseCount?
>>>>>
>>>>> I'm pretty sure that we did not have any issues with this before.
>>>>>
>>>>> Thanks,
>>>>> Michael
>>>>
>>>>
>>>
>>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by michael dürr <du...@gmail.com>.
Thanks to all of you for your advice on using the terms query! I wasn't
aware of this syntax until now.
Anyways it would be good to know whether I hit a bug or not.
Are my example queries probably rewritten to something that has more
boolean clauses?
If so, why doesn't that apply to the query for the unique key field?
Maybe someone can give some insights here?
Thanks,
Michael
On Thu, Dec 1, 2022 at 7:45 PM Kevin Risden <kr...@apache.org> wrote:
>
> https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
>
> The "!{terms ..." syntax is short for a query parser. Its a terms query
> parser and as Jan said its way more efficient than boolean clauses for a
> list of terms.
> Kevin Risden
>
>
>
> On Thu, Dec 1, 2022 at 1:04 PM Thomas Heigl <th...@umschalt.com> wrote:
>
> > Hi Jan,
> >
> > We ran into the same issue. Terms queries sound like the ideal solution
> for
> > our use case, but I couldn't find any documentation on the {!terms}
> syntax.
> > Is there anything in the official docs?
> >
> > Best,
> >
> > Thomas
> >
> > On Thu, Dec 1, 2022 at 2:09 PM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >
> > > Have you tried using Terms Query? It is much more efficient than many
> > > boolean should clauses
> > >
> > > ?q={!terms f=id}1 2 3 4...1025
> > >
> > > Jan
> > >
> > > > 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
> > > >
> > > > Hi,
> > > >
> > > > today we updated solr to version 9.1 (lucene version 9.3)
> > > > Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> > > > setting for maxClauseCount is 1024
> > > > I played around a lot and could trace it down to this:
> > > >
> > > > * I built an index from scratch with two fields (id is unique key)
> and
> > > > luceneMatchVersion 9.3:
> > > >
> > > > <field name="id" type="string_dv" indexed="true" stored="true"
> > > > multiValued="false" required="true"/>
> > > > <field name="createdById" type="p_long_dv" indexed="true"
> > stored="false"
> > > > multiValued="false" />
> > > >
> > > > <fieldType name="string_dv" class="solr.StrField"
> > sortMissingLast="true"
> > > > omitNorms="true" docValues="true" />
> > > > fieldType name="p_long_dv" class="solr.LongPointField"
> docValues="true"
> > > > omitNorms="true" />
> > > >
> > > > As expected this works (the dots(...) represent the complete set of
> > > numbers
> > > > up to 1024):
> > > >
> > > > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
> ...
> > > 1024)
> > > >
> > > > And this fails:
> > > >
> > > > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3
> ...
> > > 1025)
> > > >
> > > > But when I use the other field (categoryId) this fails:
> > > >
> > > > curl -XGET
> http://localhost:8983/solr/myindex/select?q=+categoryId:(1
> > 2
> > > 3
> > > > ... 1024)
> > > >
> > > > It works until 512 and starts failing from 513 clauses
> > > >
> > > > No difference when doing it like this:
> > > >
> > > > curl -XGET
> http://localhost:8983/solr/myindex/select?q=+(categoryId:1
> > > > categoryId:2 ... categoryId:1024)
> > > >
> > > > Am I misunderstanding the limit maxClauseCount?
> > > >
> > > > I'm pretty sure that we did not have any issues with this before.
> > > >
> > > > Thanks,
> > > > Michael
> > >
> > >
> >
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Kevin Risden <kr...@apache.org>.
https://solr.apache.org/guide/solr/latest/query-guide/other-parsers.html#terms-query-parser
The "!{terms ..." syntax is short for a query parser. Its a terms query
parser and as Jan said its way more efficient than boolean clauses for a
list of terms.
Kevin Risden
On Thu, Dec 1, 2022 at 1:04 PM Thomas Heigl <th...@umschalt.com> wrote:
> Hi Jan,
>
> We ran into the same issue. Terms queries sound like the ideal solution for
> our use case, but I couldn't find any documentation on the {!terms} syntax.
> Is there anything in the official docs?
>
> Best,
>
> Thomas
>
> On Thu, Dec 1, 2022 at 2:09 PM Jan Høydahl <ja...@cominvent.com> wrote:
>
> > Have you tried using Terms Query? It is much more efficient than many
> > boolean should clauses
> >
> > ?q={!terms f=id}1 2 3 4...1025
> >
> > Jan
> >
> > > 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
> > >
> > > Hi,
> > >
> > > today we updated solr to version 9.1 (lucene version 9.3)
> > > Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> > > setting for maxClauseCount is 1024
> > > I played around a lot and could trace it down to this:
> > >
> > > * I built an index from scratch with two fields (id is unique key) and
> > > luceneMatchVersion 9.3:
> > >
> > > <field name="id" type="string_dv" indexed="true" stored="true"
> > > multiValued="false" required="true"/>
> > > <field name="createdById" type="p_long_dv" indexed="true"
> stored="false"
> > > multiValued="false" />
> > >
> > > <fieldType name="string_dv" class="solr.StrField"
> sortMissingLast="true"
> > > omitNorms="true" docValues="true" />
> > > fieldType name="p_long_dv" class="solr.LongPointField" docValues="true"
> > > omitNorms="true" />
> > >
> > > As expected this works (the dots(...) represent the complete set of
> > numbers
> > > up to 1024):
> > >
> > > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ...
> > 1024)
> > >
> > > And this fails:
> > >
> > > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ...
> > 1025)
> > >
> > > But when I use the other field (categoryId) this fails:
> > >
> > > curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1
> 2
> > 3
> > > ... 1024)
> > >
> > > It works until 512 and starts failing from 513 clauses
> > >
> > > No difference when doing it like this:
> > >
> > > curl -XGET http://localhost:8983/solr/myindex/select?q=+(categoryId:1
> > > categoryId:2 ... categoryId:1024)
> > >
> > > Am I misunderstanding the limit maxClauseCount?
> > >
> > > I'm pretty sure that we did not have any issues with this before.
> > >
> > > Thanks,
> > > Michael
> >
> >
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Thomas Heigl <th...@umschalt.com>.
Hi Jan,
We ran into the same issue. Terms queries sound like the ideal solution for
our use case, but I couldn't find any documentation on the {!terms} syntax.
Is there anything in the official docs?
Best,
Thomas
On Thu, Dec 1, 2022 at 2:09 PM Jan Høydahl <ja...@cominvent.com> wrote:
> Have you tried using Terms Query? It is much more efficient than many
> boolean should clauses
>
> ?q={!terms f=id}1 2 3 4...1025
>
> Jan
>
> > 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
> >
> > Hi,
> >
> > today we updated solr to version 9.1 (lucene version 9.3)
> > Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> > setting for maxClauseCount is 1024
> > I played around a lot and could trace it down to this:
> >
> > * I built an index from scratch with two fields (id is unique key) and
> > luceneMatchVersion 9.3:
> >
> > <field name="id" type="string_dv" indexed="true" stored="true"
> > multiValued="false" required="true"/>
> > <field name="createdById" type="p_long_dv" indexed="true" stored="false"
> > multiValued="false" />
> >
> > <fieldType name="string_dv" class="solr.StrField" sortMissingLast="true"
> > omitNorms="true" docValues="true" />
> > fieldType name="p_long_dv" class="solr.LongPointField" docValues="true"
> > omitNorms="true" />
> >
> > As expected this works (the dots(...) represent the complete set of
> numbers
> > up to 1024):
> >
> > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ...
> 1024)
> >
> > And this fails:
> >
> > curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ...
> 1025)
> >
> > But when I use the other field (categoryId) this fails:
> >
> > curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1 2
> 3
> > ... 1024)
> >
> > It works until 512 and starts failing from 513 clauses
> >
> > No difference when doing it like this:
> >
> > curl -XGET http://localhost:8983/solr/myindex/select?q=+(categoryId:1
> > categoryId:2 ... categoryId:1024)
> >
> > Am I misunderstanding the limit maxClauseCount?
> >
> > I'm pretty sure that we did not have any issues with this before.
> >
> > Thanks,
> > Michael
>
>
Re: Wired behavior of maxClauseCount restriction since upgrading to solr 9.1
Posted by Jan Høydahl <ja...@cominvent.com>.
Have you tried using Terms Query? It is much more efficient than many boolean should clauses
?q={!terms f=id}1 2 3 4...1025
Jan
> 1. des. 2022 kl. 13:27 skrev michael dürr <du...@gmail.com>:
>
> Hi,
>
> today we updated solr to version 9.1 (lucene version 9.3)
> Since then we noticed plenty of TooManyNestedClauses in the logs. Our
> setting for maxClauseCount is 1024
> I played around a lot and could trace it down to this:
>
> * I built an index from scratch with two fields (id is unique key) and
> luceneMatchVersion 9.3:
>
> <field name="id" type="string_dv" indexed="true" stored="true"
> multiValued="false" required="true"/>
> <field name="createdById" type="p_long_dv" indexed="true" stored="false"
> multiValued="false" />
>
> <fieldType name="string_dv" class="solr.StrField" sortMissingLast="true"
> omitNorms="true" docValues="true" />
> fieldType name="p_long_dv" class="solr.LongPointField" docValues="true"
> omitNorms="true" />
>
> As expected this works (the dots(...) represent the complete set of numbers
> up to 1024):
>
> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ... 1024)
>
> And this fails:
>
> curl -XGET http://localhost:8983/solr/myindex/select?q=+id:(1 2 3 ... 1025)
>
> But when I use the other field (categoryId) this fails:
>
> curl -XGET http://localhost:8983/solr/myindex/select?q=+categoryId:(1 2 3
> ... 1024)
>
> It works until 512 and starts failing from 513 clauses
>
> No difference when doing it like this:
>
> curl -XGET http://localhost:8983/solr/myindex/select?q=+(categoryId:1
> categoryId:2 ... categoryId:1024)
>
> Am I misunderstanding the limit maxClauseCount?
>
> I'm pretty sure that we did not have any issues with this before.
>
> Thanks,
> Michael