You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dmitry Kan <so...@gmail.com> on 2013/04/24 14:08:28 UTC

Fwd: [solr 3.4] anomaly during distributed facet query with 102 shards

Hello list,

We deal with an anomaly when doing a distributed facet query against 102
shards.

The problem manifests itself in both the frontend solr (router) and a
shard. Each time the request is executed, always different shard is
affected (at random, hence the "anomaly").

The query is: http://router_host:router_port
/solr/select?q=test&facet=true&facet.field=field_of_type_long&facet.limit=1330&facet.mincount=1&rows=1&facet.sort=index&facet.zeros=false&facet.offset=0
I have omitted the shards parameter.

The router log:

request: http://10.155.244.181:9150/solr/select
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
        at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421)
        at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Notice the port of a shard, that is affected. That port changes all the
time, even for the same request
The log entry is prepended with lines:

SEVERE: org.apache.solr.common.SolrException: Internal Server Error

Internal Server Error

(they are not in the pastebin link)

The shard log:

Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
        at java.io.StringReader.<init>(StringReader.java:50)
        at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
        at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
        at org.apache.solr.search.QParser.getQuery(QParser.java:142)
        at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
        at java.lang.Thread.run(Thread.java:722)

Apr 24, 2013 11:08:49 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={} status=500 QTime=2
Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
        at java.io.StringReader.<init>(StringReader.java:50)
        at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
        at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
        at org.apache.solr.search.QParser.getQuery(QParser.java:142)
        at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
        at java.lang.Thread.run(Thread.java:722)


Essentially, for some reason, the shard has received an empty request. The
hypothesis we have is: the router sends two queries, of which second is
empty. The first one being the original request, the second being the
request with doc ids. Is this correct?

GET limitations should not be affecting here, as the router sends a
distributed query via POST.

In the attempt to solve the issue we have tried modifying the shard timeout
settings on the router. Currently set values are:
shard-connection-timeout=300000&shard-socket-timeout=300000

This didn't help though.

What could be tried else?

Thanks,

Dmitry

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

Posted by Dmitry Kan <so...@gmail.com>.
Hi,

1. Ruled out possibility to test 4.2.1 router against 3.4 shard farm for
obvious reasons (java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format).

2. Tried jetty, but same result.


On Thu, Apr 25, 2013 at 5:16 PM, Dmitry Kan <so...@gmail.com> wrote:

> Thanks, Yonik. Yes, I supposed that. We are in the pre-release phase, so
> we have the pressure.
>
> Solr 3.4.
>
> Would setting up 4.2.1 router work with 3.4 shards?
> On 25 Apr 2013 17:11, "Yonik Seeley" <yo...@lucidworks.com> wrote:
>
>> On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan <so...@gmail.com> wrote:
>> > Are there any distrib facet gurus on the list? I would be ready to try
>> > sensible ideas, including on the source code level, if someone of you
>> could
>> > give me a hand.
>>
>> The Lucene/Solr Revolution conference is coming up next week, so I
>> think many are busy creating their presentations.
>> What version of Solr are you using?  Have you tried using a newer
>> version?  Is it reproducable with a smaller cluster?  If so, you could
>> try using the included Jetty server instead of Tomcat to rule out that
>> factor.
>>
>> -Yonik
>> http://lucidworks.com
>>
>

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

Posted by Dmitry Kan <so...@gmail.com>.
Thanks, Yonik. Yes, I supposed that. We are in the pre-release phase, so we
have the pressure.

Solr 3.4.

Would setting up 4.2.1 router work with 3.4 shards?
On 25 Apr 2013 17:11, "Yonik Seeley" <yo...@lucidworks.com> wrote:

> On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan <so...@gmail.com> wrote:
> > Are there any distrib facet gurus on the list? I would be ready to try
> > sensible ideas, including on the source code level, if someone of you
> could
> > give me a hand.
>
> The Lucene/Solr Revolution conference is coming up next week, so I
> think many are busy creating their presentations.
> What version of Solr are you using?  Have you tried using a newer
> version?  Is it reproducable with a smaller cluster?  If so, you could
> try using the included Jetty server instead of Tomcat to rule out that
> factor.
>
> -Yonik
> http://lucidworks.com
>

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

Posted by Yonik Seeley <yo...@lucidworks.com>.
On Thu, Apr 25, 2013 at 8:32 AM, Dmitry Kan <so...@gmail.com> wrote:
> Are there any distrib facet gurus on the list? I would be ready to try
> sensible ideas, including on the source code level, if someone of you could
> give me a hand.

The Lucene/Solr Revolution conference is coming up next week, so I
think many are busy creating their presentations.
What version of Solr are you using?  Have you tried using a newer
version?  Is it reproducable with a smaller cluster?  If so, you could
try using the included Jetty server instead of Tomcat to rule out that
factor.

-Yonik
http://lucidworks.com

Re: [solr 3.4] anomaly during distributed facet query with 102 shards

Posted by Dmitry Kan <so...@gmail.com>.
Are there any distrib facet gurus on the list? I would be ready to try
sensible ideas, including on the source code level, if someone of you could
give me a hand.

Dmitry


On Wed, Apr 24, 2013 at 3:08 PM, Dmitry Kan <so...@gmail.com> wrote:

> Hello list,
>
> We deal with an anomaly when doing a distributed facet query against 102
> shards.
>
> The problem manifests itself in both the frontend solr (router) and a
> shard. Each time the request is executed, always different shard is
> affected (at random, hence the "anomaly").
>
> The query is: http://router_host:router_port
> /solr/select?q=test&facet=true&facet.field=field_of_type_long&facet.limit=1330&facet.mincount=1&rows=1&facet.sort=index&facet.zeros=false&facet.offset=0
> I have omitted the shards parameter.
>
> The router log:
>
> request: http://10.155.244.181:9150/solr/select
>         at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
>         at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
>         at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421)
>         at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
>
> Notice the port of a shard, that is affected. That port changes all the
> time, even for the same request
> The log entry is prepended with lines:
>
> SEVERE: org.apache.solr.common.SolrException: Internal Server Error
>
> Internal Server Error
>
> (they are not in the pastebin link)
>
> The shard log:
>
> Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
>         at java.io.StringReader.<init>(StringReader.java:50)
>         at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
>         at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
>         at org.apache.solr.search.QParser.getQuery(QParser.java:142)
>         at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
>         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>         at java.lang.Thread.run(Thread.java:722)
>
> Apr 24, 2013 11:08:49 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select params={} status=500 QTime=2
> Apr 24, 2013 11:08:49 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
>         at java.io.StringReader.<init>(StringReader.java:50)
>         at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:203)
>         at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:80)
>         at org.apache.solr.search.QParser.getQuery(QParser.java:142)
>         at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:81)
>         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>         at java.lang.Thread.run(Thread.java:722)
>
>
> Essentially, for some reason, the shard has received an empty request. The
> hypothesis we have is: the router sends two queries, of which second is
> empty. The first one being the original request, the second being the
> request with doc ids. Is this correct?
>
> GET limitations should not be affecting here, as the router sends a
> distributed query via POST.
>
> In the attempt to solve the issue we have tried modifying the shard
> timeout settings on the router. Currently set values are:
> shard-connection-timeout=300000&shard-socket-timeout=300000
>
> This didn't help though.
>
> What could be tried else?
>
> Thanks,
>
> Dmitry
>
>