You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mikhail Khludnev (JIRA)" <ji...@apache.org> on 2015/11/19 10:46:11 UTC
[jira] [Comment Edited] (SOLR-7606) ToParentBlockJoinQuery fails
with AIOOBE under certain circumstances
[ https://issues.apache.org/jira/browse/SOLR-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007506#comment-15007506 ]
Mikhail Khludnev edited comment on SOLR-7606 at 11/19/15 9:45 AM:
------------------------------------------------------------------
attaching [^SOLR-7606.patch]:
* it fixes Solr's {{BlockJoinParentQParser.java}} only, there is no Lucene changes, so far
* {{BlockJoinParentQParser.BitDocIdSetFilterWrapper}} remembers top readers which it was checked against.
** it's necessary because {{NoOpRegenerator}} can transfer it across top-level reader
** a check flag could be stored by every segments instead top-level reader, but I judge about a code perfomance by my gut feeling
* thus, it lazily (on every commit) invokes {{CheckJoinIndex}} from LUCENE-6589, I think it's affordable (see the point above).
* test reproduces absence of last "boundary" parent in a segment, and no parent hits cases
* note, that such check is possible for the root BJQ query, deeper levels seem a danger zone (btw, now it counts any enclosing queries, but have to count only bjq's TODO)
if you second this approach, I'd also like *to do* here:
- cover deletes case with a test
- also cover \{!child}
- it should be possible to use "just a segment filter" hack, when \{!parent} is invoked without subordinate clause
- how does it work if second level bitset is broken on grandchidren search eg {!parent}...{!parent which="type:child -id:333"}...
was (Author: mkhludnev):
attaching [^SOLR-7606.patch]:
* it fixes Solr's {{BlockJoinParentQParser.java}} only, there is no Lucene changes, so far
* {{BlockJoinParentQParser.BitDocIdSetFilterWrapper}} remembers top readers which it was checked against.
** it's necessary because {{NoOpRegenerator}} can transfer it across top-level reader
** a check flag could be stored by every segments instead top-level reader, but I judge about a code perfomance by my gut feeling
* thus, it lazily (on every commit) invokes {{CheckJoinIndex}} from LUCENE-6589, I think it's affordable (see the point above).
* test reproduces absence of last "boundary" parent in a segment, and no parent hits cases
* note, that such check is possible for the root BJQ query, deeper levels seem a danger zone (btw, now it counts any enclosing queries, but have to count only bjq's TODO)
if you second this approach, I'd also like to do right here:
- cover deletes case with a test
- also cover \{!child}
- it should be possible to use "just a segment filter" hack, when \{!parent} is invoked without subordinate clause
> ToParentBlockJoinQuery fails with AIOOBE under certain circumstances
> --------------------------------------------------------------------
>
> Key: SOLR-7606
> URL: https://issues.apache.org/jira/browse/SOLR-7606
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.10.4
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Attachments: LUCENE-6512.patch, LUCENE-6512.patch, SOLR-7606.patch, SOLR-7606.patch
>
>
> I had a customer using BlockJoin with Solr. He executed a block join query and the following appeared in Solr's logs:
> {noformat}
> 28 May 2015 17:19:20 ERROR (SolrException.java:131) - java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.lucene.codecs.lucene40.BitVector.get(BitVector.java:149)
> at org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinScorer.nextDoc(ToParentBlockJoinQuery.java:293)
> at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)
> at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
> at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:209)
> at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1619)
> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1433)
> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:514)
> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:484)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I debugged this stuff and found out when this happens:
> The last block of documents was not followed by a parent. If one of the child documents without a parent at the end of the index match the inner query, scorer calls nextSetBit() to find next parent document. This returns -1. There is an assert afterwards that checks for -1, but in production code, this is of course never executed.
> If the index has deletetions the false -1 is passed to acceptDocs and then triggers the above problem.
> We should change the assert to another IllegalStateException() which is used to notify the user if the orthogonality is broken. By that the user gets the information that his index is broken and contains child documents without a parent at the very end of a segment.
> I have seen this on 4.10.4. Maybe thats already fixed in 5.0, but I just open this here for investigation. This was clearly a problem in the index, but due to Solr's buggy implementation of parent/child documents (you have to set the parent flag in contrast to Elasticsearch on your own - which is stupid!!!) this was not detected at indexing time. We should open an issue in Solr to fix this bad behaviour and make solr automatically add the parent field (it only adds a "_root_" field automatically, maybe it should also add a "_parent_" field automatically).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org