You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dmitry Kan <dm...@gmail.com> on 2012/04/11 14:16:50 UTC
solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Hello,
Hopefully this question is not too complex to handle, but I'm currently
stuck with it.
We have a system with nTiers, that is:
Solr front base ---> Solr front --> shards
Inside QueryComponent there is a method createRetrieveDocs(ResponseBuilder
rb) which collects doc ids of each shard and sends them in different
queries using the ids parameter:
[code]
sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
[/code]
This actually produces NPE (same as in
https://issues.apache.org/jira/browse/SOLR-1477) in the first tier, because
Solr front (on the second tier) fails to process such a query. I have tried
to fix this by using a unique field with a value of ids ORed (the following
code substitutes the code above):
[code]
StringBuffer idsORed = new StringBuffer();
for (Iterator<String> iterator = ids.iterator(); iterator.hasNext();
) {
String next = iterator.next();
if (iterator.hasNext()) {
idsORed.append(next).append(" OR ");
} else {
idsORed.append(next);
}
}
sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
idsORed.toString());
[/code]
This works perfectly if for rows=n there is n or less hits from a
distributed query. However, if there are more than 2*n hits, the querying
fails with an NPE in a completely different component, which is
HighlightComponent (highlights are requested in the same query with
hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
SEVERE: java.lang.NullPointerException
at
org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
It sounds like the ids of documents somehow get shuffled and the
instruction (only a hypothesis)
[code]
ShardDoc sdoc = rb.resultIds.get(id);
[/code]
returns sdoc=null, which causes the next line of code to fail with an NPE:
[code]
int idx = sdoc.positionInResponse;
[/code]
Am I missing anything? Can something be done for solving this issue?
Thanks.
--
Regards,
Dmitry Kan
Re: solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Posted by Dmitry Kan <dm...@gmail.com>.
Mikhail,
Thanks for sharing your thoughts. Yes I have tried checking for NULL and
the entire chain of queries between tiers seems to work. But I suspect,
that some docs will be missing. In principle, unless there is an
OutOfMemory or a shard down, the doc ids should be retrieving valid
documents. So this is just a design, as Yonik pointed out.
I would be willing to contribute a patch, it is just an issue of
understanding what exactly should be fixed in the architecture, and I
suspect it isn't a small change..
Dmitry
On Thu, Apr 12, 2012 at 9:22 PM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Dmitry,
>
> The last NPE in HighlightingComponent is just a sad coding issue.
> few rows later we can see that developer expected to have some docs not
> found
> // remove nulls in case not all docs were able to be retrieved
> rb.rsp.add("highlighting", SolrPluginUtils.removeNulls(new
> SimpleOrderedMap(arr)));
> But as you already know he forgot to check if(sdoc!=null){.
> Is there anything that stopping you from contributing the patch, beside of
> the lack of time, of course?
>
> about the core issue I can't get into it and, particularly, how the using
> disjunction query in place of IDS can help you. Could you please provide
> more detailed info like stacktraces, etc. Btw, have you checked trunk for
> your case?
>
> On Thu, Apr 12, 2012 at 7:08 PM, Dmitry Kan <dm...@gmail.com> wrote:
>
> > Can anyone help me out with this? Is this too complicated / unclear? I
> > could share more detail if needed.
> >
> > On Wed, Apr 11, 2012 at 3:16 PM, Dmitry Kan <dm...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Hopefully this question is not too complex to handle, but I'm currently
> > > stuck with it.
> > >
> > > We have a system with nTiers, that is:
> > >
> > > Solr front base ---> Solr front --> shards
> > >
> > > Inside QueryComponent there is a method
> > createRetrieveDocs(ResponseBuilder
> > > rb) which collects doc ids of each shard and sends them in different
> > > queries using the ids parameter:
> > >
> > > [code]
> > > sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
> > > [/code]
> > >
> > > This actually produces NPE (same as in
> > > https://issues.apache.org/jira/browse/SOLR-1477) in the first tier,
> > > because Solr front (on the second tier) fails to process such a query.
> I
> > > have tried to fix this by using a unique field with a value of ids ORed
> > > (the following code substitutes the code above):
> > >
> > > [code]
> > > StringBuffer idsORed = new StringBuffer();
> > > for (Iterator<String> iterator = ids.iterator();
> > iterator.hasNext();
> > > ) {
> > > String next = iterator.next();
> > >
> > > if (iterator.hasNext()) {
> > > idsORed.append(next).append(" OR ");
> > > } else {
> > > idsORed.append(next);
> > > }
> > > }
> > >
> > > sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
> > > idsORed.toString());
> > > [/code]
> > >
> > > This works perfectly if for rows=n there is n or less hits from a
> > > distributed query. However, if there are more than 2*n hits, the
> querying
> > > fails with an NPE in a completely different component, which is
> > > HighlightComponent (highlights are requested in the same query with
> > >
> >
> hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
> > >
> > > SEVERE: java.lang.NullPointerException
> > > at
> > >
> >
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
> > > at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > > at
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > > at
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > at
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > at
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > at
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> > > at
> > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > > at java.lang.Thread.run(Thread.java:619)
> > >
> > > It sounds like the ids of documents somehow get shuffled and the
> > > instruction (only a hypothesis)
> > >
> > > [code]
> > > ShardDoc sdoc = rb.resultIds.get(id);
> > > [/code]
> > >
> > > returns sdoc=null, which causes the next line of code to fail with an
> > NPE:
> > >
> > > [code]
> > > int idx = sdoc.positionInResponse;
> > > [/code]
> > >
> > > Am I missing anything? Can something be done for solving this issue?
> > >
> > > Thanks.
> > >
> > > --
> > > Regards,
> > >
> > > Dmitry Kan
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> gedel@yandex.ru
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
--
Regards,
Dmitry Kan
Re: solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Dmitry,
The last NPE in HighlightingComponent is just a sad coding issue.
few rows later we can see that developer expected to have some docs not
found
// remove nulls in case not all docs were able to be retrieved
rb.rsp.add("highlighting", SolrPluginUtils.removeNulls(new
SimpleOrderedMap(arr)));
But as you already know he forgot to check if(sdoc!=null){.
Is there anything that stopping you from contributing the patch, beside of
the lack of time, of course?
about the core issue I can't get into it and, particularly, how the using
disjunction query in place of IDS can help you. Could you please provide
more detailed info like stacktraces, etc. Btw, have you checked trunk for
your case?
On Thu, Apr 12, 2012 at 7:08 PM, Dmitry Kan <dm...@gmail.com> wrote:
> Can anyone help me out with this? Is this too complicated / unclear? I
> could share more detail if needed.
>
> On Wed, Apr 11, 2012 at 3:16 PM, Dmitry Kan <dm...@gmail.com> wrote:
>
> > Hello,
> >
> > Hopefully this question is not too complex to handle, but I'm currently
> > stuck with it.
> >
> > We have a system with nTiers, that is:
> >
> > Solr front base ---> Solr front --> shards
> >
> > Inside QueryComponent there is a method
> createRetrieveDocs(ResponseBuilder
> > rb) which collects doc ids of each shard and sends them in different
> > queries using the ids parameter:
> >
> > [code]
> > sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
> > [/code]
> >
> > This actually produces NPE (same as in
> > https://issues.apache.org/jira/browse/SOLR-1477) in the first tier,
> > because Solr front (on the second tier) fails to process such a query. I
> > have tried to fix this by using a unique field with a value of ids ORed
> > (the following code substitutes the code above):
> >
> > [code]
> > StringBuffer idsORed = new StringBuffer();
> > for (Iterator<String> iterator = ids.iterator();
> iterator.hasNext();
> > ) {
> > String next = iterator.next();
> >
> > if (iterator.hasNext()) {
> > idsORed.append(next).append(" OR ");
> > } else {
> > idsORed.append(next);
> > }
> > }
> >
> > sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
> > idsORed.toString());
> > [/code]
> >
> > This works perfectly if for rows=n there is n or less hits from a
> > distributed query. However, if there are more than 2*n hits, the querying
> > fails with an NPE in a completely different component, which is
> > HighlightComponent (highlights are requested in the same query with
> >
> hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
> >
> > SEVERE: java.lang.NullPointerException
> > at
> >
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> > at
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > at
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> > at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > at java.lang.Thread.run(Thread.java:619)
> >
> > It sounds like the ids of documents somehow get shuffled and the
> > instruction (only a hypothesis)
> >
> > [code]
> > ShardDoc sdoc = rb.resultIds.get(id);
> > [/code]
> >
> > returns sdoc=null, which causes the next line of code to fail with an
> NPE:
> >
> > [code]
> > int idx = sdoc.positionInResponse;
> > [/code]
> >
> > Am I missing anything? Can something be done for solving this issue?
> >
> > Thanks.
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
>
>
> --
> Regards,
>
> Dmitry Kan
>
--
Sincerely yours
Mikhail Khludnev
gedel@yandex.ru
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Posted by Dmitry Kan <dm...@gmail.com>.
Can anyone help me out with this? Is this too complicated / unclear? I
could share more detail if needed.
On Wed, Apr 11, 2012 at 3:16 PM, Dmitry Kan <dm...@gmail.com> wrote:
> Hello,
>
> Hopefully this question is not too complex to handle, but I'm currently
> stuck with it.
>
> We have a system with nTiers, that is:
>
> Solr front base ---> Solr front --> shards
>
> Inside QueryComponent there is a method createRetrieveDocs(ResponseBuilder
> rb) which collects doc ids of each shard and sends them in different
> queries using the ids parameter:
>
> [code]
> sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
> [/code]
>
> This actually produces NPE (same as in
> https://issues.apache.org/jira/browse/SOLR-1477) in the first tier,
> because Solr front (on the second tier) fails to process such a query. I
> have tried to fix this by using a unique field with a value of ids ORed
> (the following code substitutes the code above):
>
> [code]
> StringBuffer idsORed = new StringBuffer();
> for (Iterator<String> iterator = ids.iterator(); iterator.hasNext();
> ) {
> String next = iterator.next();
>
> if (iterator.hasNext()) {
> idsORed.append(next).append(" OR ");
> } else {
> idsORed.append(next);
> }
> }
>
> sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
> idsORed.toString());
> [/code]
>
> This works perfectly if for rows=n there is n or less hits from a
> distributed query. However, if there are more than 2*n hits, the querying
> fails with an NPE in a completely different component, which is
> HighlightComponent (highlights are requested in the same query with
> hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):
>
> SEVERE: java.lang.NullPointerException
> at
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:619)
>
> It sounds like the ids of documents somehow get shuffled and the
> instruction (only a hypothesis)
>
> [code]
> ShardDoc sdoc = rb.resultIds.get(id);
> [/code]
>
> returns sdoc=null, which causes the next line of code to fail with an NPE:
>
> [code]
> int idx = sdoc.positionInResponse;
> [/code]
>
> Am I missing anything? Can something be done for solving this issue?
>
> Thanks.
>
> --
> Regards,
>
> Dmitry Kan
>
--
Regards,
Dmitry Kan
Re: solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Posted by Dmitry Kan <dm...@gmail.com>.
Thanks Yonik,
This is what I expected. How big the change would be, if I'd start just
with Query and Highlight components? Did the change to QueryComponent I
made make any sense to you? It would of course mean a custom solution,
which I'm willing to contribute as a patch (in case anyone interested). To
make it part of a releasable trunk, one would most probably need to provide
some way to configure "1st tier level".
Thanks,
Dmitry
On Thu, Apr 12, 2012 at 9:34 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Wed, Apr 11, 2012 at 8:16 AM, Dmitry Kan <dm...@gmail.com> wrote:
> > We have a system with nTiers, that is:
> >
> > Solr front base ---> Solr front --> shards
>
> Although the architecture had this in mind (multi-tier), all of the
> pieces are not yet in place to allow it.
> The errors you see are a direct result of that.
>
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10
>
--
Regards,
Dmitry Kan
Re: solr 3.4 with nTiers >= 2: usage of ids param causes
NullPointerException (NPE)
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Apr 11, 2012 at 8:16 AM, Dmitry Kan <dm...@gmail.com> wrote:
> We have a system with nTiers, that is:
>
> Solr front base ---> Solr front --> shards
Although the architecture had this in mind (multi-tier), all of the
pieces are not yet in place to allow it.
The errors you see are a direct result of that.
-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10