You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Gus Heck (JIRA)" <ji...@apache.org> on 2019/05/01 14:30:00 UTC
[jira] [Commented] (SOLR-8291) NPE calling export handler when
useFilterForSortedQuery=true
[ https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831022#comment-16831022 ]
Gus Heck commented on SOLR-8291:
--------------------------------
Just hit this recently. [~ronnnnnnn]'s explanation seems to jibe with what I was finding as I prowled the code. He found the spot where the bitsets were left null which I hadn't found yet, so kudos and thanks to him there :). It looks like there have been no major changes to the logic of the code in sortDocSet since LUCENE-2831 in 2011, and the export handler code seems to only date back to 2014, so the assumptions in ExportQParserPlugin don't seem to match the previously established behavior of not calling getLeafCollector after the last matching doc. Given the history, I'd say that the assumption by ExportQParserPlugin that it can create the array of bit sets in advance based on the leaf count is the underlying flaw.
{code:java}
public TopDocsCollector getTopDocsCollector(int len,
QueryCommand cmd,
IndexSearcher searcher) throws IOException {
int leafCount = searcher.getTopReaderContext().leaves().size();
FixedBitSet[] sets = new FixedBitSet[leafCount];
return new ExportCollector(sets);
} {code}
This makes the problem specific to the Export handler code path, and general search path stuff like sortDocSet() probably should be left alone. A quick browse into half a dozen other TopDocsCollectors don't show any use of the leaf count similar to this, but I did not do an exhaustive search.
Two fixes come to mind within the export handler code, the patch supplied by Ron, tweaked to add the null check to the loop condition, or an else block break to avoid useless iterations. The second is for ExportQParserPlugin to populate with empty bitsets, which is unattractive for memory reasons, but solves the problem at the source.
The first solution (a version of Ron's patch) amounts to ExportWriter being more tolerant in what it accepts and short circuiting earlier, so I think that's the preferable path vs creating more objects for GC. I'll be testing such a patch on the system that exposed this for me soon.
> NPE calling export handler when useFilterForSortedQuery=true
> ------------------------------------------------------------
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 5.2.1
> Reporter: Ray
> Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the following element:
> {code}
> <useFilterForSortedQuery>true</useFilterForSortedQuery>
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the /export handler.
> See the comments near the end of the ticket for a potential work around if this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.<init>(BitSetIterator.java:58)
> at org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org