You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Michael <so...@gmail.com> on 2010/01/07 22:17:39 UTC

NPE when sharding multiple layers

Hi all,

I've got an index split across 28 cores -- 4 cores on each of 7 boxes
(multiple cores per box in order to use more of its CPUs.)

When I configure a "toplevel" core to fan out to all 28 index cores,
it works, but is slower than I'd have expected:
     Toplevel core ==> all 28 index cores

In case it is the aggregation of 28 shards that is slow, I wanted to
try 2 layers of sharding.  I changed the toplevel core to shard to 1
"midlevel" core per box, which in turn shards to the 4 index cores on
localhost:
     Toplevel core ==> 7 midlevel cores, 1 per box ==> 4 localhost index cores

If I search for *:*, this works.  If I search for an actual
field:value, the midlevel cores throw an NPE.

I am configuring toplevel and midlevel cores' &shards= parameters via
default values in their solrconfigs, so my request URL just looks like
host/solr/toplevel/select/&q=field:value.

Is this a known bug, or am I just doing something wrong?

Thanks in advance!
- Michael

PS: The NPE, which is thrown by the midlevel cores:

Jan 7, 2010 4:01:02 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
        at org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardDoc.java:210)
        at org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardDoc.java:134)
        at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:255)
        at org.apache.lucene.util.PriorityQueue.put(PriorityQueue.java:114)
        at org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:156)
        at org.apache.lucene.util.PriorityQueue.insert(PriorityQueue.java:141)
        at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:445)
        at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
        at java.lang.Thread.run(Thread.java:619)

Re: NPE when sharding multiple layers

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jan 7, 2010 at 4:33 PM, Michael <so...@gmail.com> wrote:
> Does "not supported" mean "we can't guarantee whether it will work or
> not", or "you may be able to figure it out on your own"?

Not implemented, and not expected to work.
For example, some info such as sortFieldValues would need to be merged
and returned as is done for leaf requests.  There are probably other
little things like that, but  I can't list them off the top of my
head.

-Yonik
http://www.lucidimagination.com

Re: NPE when sharding multiple layers

Posted by Michael <so...@gmail.com>.
Thanks, Yonik.

Does "not supported" mean "we can't guarantee whether it will work or
not", or "you may be able to figure it out on your own"?  Apparently I
am able to get *some* queries through, just not those that pass
through the fieldtype that i really need (a complex analyzer).  When I
search for foo:value where foo is an analyzer that uses

  StandardTokenizer
  LowerCaseFilter
  WordDelimeterFilter
  TrimFilter

I *don't* get an NPE.

Thanks,
Michael

On Thu, Jan 7, 2010 at 4:25 PM, Yonik Seeley <yo...@lucidimagination.com> wrote:
> On Thu, Jan 7, 2010 at 4:17 PM, Michael <so...@gmail.com> wrote:
>>  I wanted to try 2 layers of sharding.
>
> Distrib search was written with multi-level in mind, but it's not supported yet.
>
> -Yonik
> http://www.lucidimagination.com
>

Re: NPE when sharding multiple layers

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jan 7, 2010 at 4:17 PM, Michael <so...@gmail.com> wrote:
>  I wanted to try 2 layers of sharding.

Distrib search was written with multi-level in mind, but it's not supported yet.

-Yonik
http://www.lucidimagination.com