You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Pawel Rog <pa...@gmail.com> on 2014/02/17 13:07:59 UTC
Solr cloud hangs
Hi,
I have quite annoying problem with Solr cloud. I have a cluster with 8
shards and with 2 replicas in each. (Solr 4.6.1)
After some time cluster doesn't respond to any update requests. Restarting
the cluster nodes doesn't help.
There are a lot of such stack traces (waiting for very long time):
- sun.misc.Unsafe.park(Native Method)
- java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
-
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
-
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
-
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
-
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
- java.lang.Thread.run(Thread.java:722)
Do you have any idea where can I look for?
--
Pawel
Re: Solr cloud hangs
Posted by Pawel Rog <pa...@gmail.com>.
There are also many errors in solr log like that one:
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for
connection from pool
at
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
at
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
Pawel
On Mon, Feb 17, 2014 at 8:01 PM, Pawel Rog <pa...@gmail.com> wrote:
> Hi,
> Here is the whole stack trace: https://gist.github.com/anonymous/9056783
>
> --
> Pawel
>
>
> On Mon, Feb 17, 2014 at 4:53 PM, Mark Miller <ma...@gmail.com>wrote:
>
>> Can you share the full stack trace dump?
>>
>> - Mark
>>
>> http://about.me/markrmiller
>>
>> On Feb 17, 2014, at 7:07 AM, Pawel Rog <pa...@gmail.com> wrote:
>>
>> > Hi,
>> > I have quite annoying problem with Solr cloud. I have a cluster with 8
>> > shards and with 2 replicas in each. (Solr 4.6.1)
>> > After some time cluster doesn't respond to any update requests.
>> Restarting
>> > the cluster nodes doesn't help.
>> >
>> > There are a lot of such stack traces (waiting for very long time):
>> >
>> >
>> > - sun.misc.Unsafe.park(Native Method)
>> > -
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> > -
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>> > -
>> >
>> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > -
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > -
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > -
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > - java.lang.Thread.run(Thread.java:722)
>> >
>> >
>> > Do you have any idea where can I look for?
>> >
>> > --
>> > Pawel
>>
>>
>
Re: Solr cloud hangs
Posted by Pawel Rog <pa...@gmail.com>.
Hi,
Here is the whole stack trace: https://gist.github.com/anonymous/9056783
--
Pawel
On Mon, Feb 17, 2014 at 4:53 PM, Mark Miller <ma...@gmail.com> wrote:
> Can you share the full stack trace dump?
>
> - Mark
>
> http://about.me/markrmiller
>
> On Feb 17, 2014, at 7:07 AM, Pawel Rog <pa...@gmail.com> wrote:
>
> > Hi,
> > I have quite annoying problem with Solr cloud. I have a cluster with 8
> > shards and with 2 replicas in each. (Solr 4.6.1)
> > After some time cluster doesn't respond to any update requests.
> Restarting
> > the cluster nodes doesn't help.
> >
> > There are a lot of such stack traces (waiting for very long time):
> >
> >
> > - sun.misc.Unsafe.park(Native Method)
> > -
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> > -
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> > -
> >
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > -
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > -
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > -
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > - java.lang.Thread.run(Thread.java:722)
> >
> >
> > Do you have any idea where can I look for?
> >
> > --
> > Pawel
>
>
Re: Solr cloud hangs
Posted by Mark Miller <ma...@gmail.com>.
Can you share the full stack trace dump?
- Mark
http://about.me/markrmiller
On Feb 17, 2014, at 7:07 AM, Pawel Rog <pa...@gmail.com> wrote:
> Hi,
> I have quite annoying problem with Solr cloud. I have a cluster with 8
> shards and with 2 replicas in each. (Solr 4.6.1)
> After some time cluster doesn't respond to any update requests. Restarting
> the cluster nodes doesn't help.
>
> There are a lot of such stack traces (waiting for very long time):
>
>
> - sun.misc.Unsafe.park(Native Method)
> - java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> -
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> -
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> -
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> -
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> - java.lang.Thread.run(Thread.java:722)
>
>
> Do you have any idea where can I look for?
>
> --
> Pawel