You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Markey <su...@gmail.com> on 2012/04/10 21:51:25 UTC
solr hangs
Hello,
I have a solr cloud setup based on a blog (
http://outerthought.org/blog/491-ot.html) and am able to bring up the
instances and cores. But when I start indexing data (through csv update),
the core throws a out of memory exception (null:java.lang.RuntimeException:
java.lang.OutOfMemoryError: unable to create new native thread). The thread
dump from new solr ui is below:
cmdDistribExecutor-8-thread-777 (827)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bd11b79
- sun.misc.Unsafe.park(Native Method)
- java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2043)
-
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
-
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking
(ConnPoolByRoute.java:403)
-
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry
(ConnPoolByRoute.java:300)
-
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection
(ThreadSafeClientConnManager.java:224)
-
org.apache.http.impl.client.DefaultRequestDirector.execute
(DefaultRequestDirector.java:401)
-
org.apache.http.impl.client.AbstractHttpClient.execute
(AbstractHttpClient.java:820)
-
org.apache.http.impl.client.AbstractHttpClient.execute
(AbstractHttpClient.java:754)
-
org.apache.http.impl.client.AbstractHttpClient.execute
(AbstractHttpClient.java:732)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request
(HttpSolrServer.java:304)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request
(HttpSolrServer.java:209)
-
org.apache.solr.update.SolrCmdDistributor$1.call
(SolrCmdDistributor.java:320)
-
org.apache.solr.update.SolrCmdDistributor$1.call
(SolrCmdDistributor.java:301)
- java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
- java.util.concurrent.FutureTask.run(FutureTask.java:166)
-
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
- java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
- java.util.concurrent.FutureTask.run(FutureTask.java:166)
-
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1110)
-
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:603)
- java.lang.Thread.run(Thread.java:679)
Apparently I do see lots of threads like above in the thread dump. I'm
using latest build from the trunk (Apr 10th). Any insights into this issue
woudl be really helpful. Thanks a lot.
Re: solr hangs
Posted by Peter Markey <su...@gmail.com>.
Thanks for the response. I have given a size of 8gb for the instance and
has only around few thousands of documents (with 15 fields each having
small amount of data)..apparently the problem is the process (solr jetty
instance) is consuming lots of threads...one time it consumed around 50k
threads and the process maxed out the allowable thread allocated by the OS
(centos) for the process..and in the admin page is see tons of threads
under Thread Dump...it's lik solr is waiting for something....i have two
leader and replica cores/shards in two instances...and i send the documents
to one of the shard through the csv update handler...
On Wed, Apr 11, 2012 at 7:39 AM, Pawel Rog <pa...@gmail.com> wrote:
> You wrote that you can see such error "OutOfMemoryError". I had such
> problems when my caches were to big. It means that there is no more free
> memory in JVM and probably full gc starts running. How big is your Java
> heap? Maybe cache sizes in yout solr are to big according to your JVM
> settings.
>
> --
> Regards,
> Pawel
>
> On Tue, Apr 10, 2012 at 9:51 PM, Peter Markey <su...@gmail.com> wrote:
>
> > Hello,
> >
> > I have a solr cloud setup based on a blog (
> > http://outerthought.org/blog/491-ot.html) and am able to bring up the
> > instances and cores. But when I start indexing data (through csv update),
> > the core throws a out of memory exception
> (null:java.lang.RuntimeException:
> > java.lang.OutOfMemoryError: unable to create new native thread). The
> thread
> > dump from new solr ui is below:
> >
> > cmdDistribExecutor-8-thread-777 (827)
> >
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bd11b79
> >
> > - sun.misc.Unsafe.park(Native Method)
> > - java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > -
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2043)
> > -
> >
> >
> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
> > -
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking
> > (ConnPoolByRoute.java:403)
> > -
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry
> > (ConnPoolByRoute.java:300)
> > -
> >
> >
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection
> > (ThreadSafeClientConnManager.java:224)
> > -
> > org.apache.http.impl.client.DefaultRequestDirector.execute
> > (DefaultRequestDirector.java:401)
> > -
> > org.apache.http.impl.client.AbstractHttpClient.execute
> > (AbstractHttpClient.java:820)
> > -
> > org.apache.http.impl.client.AbstractHttpClient.execute
> > (AbstractHttpClient.java:754)
> > -
> > org.apache.http.impl.client.AbstractHttpClient.execute
> > (AbstractHttpClient.java:732)
> > -
> > org.apache.solr.client.solrj.impl.HttpSolrServer.request
> > (HttpSolrServer.java:304)
> > -
> > org.apache.solr.client.solrj.impl.HttpSolrServer.request
> > (HttpSolrServer.java:209)
> > -
> > org.apache.solr.update.SolrCmdDistributor$1.call
> > (SolrCmdDistributor.java:320)
> > -
> > org.apache.solr.update.SolrCmdDistributor$1.call
> > (SolrCmdDistributor.java:301)
> > - java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > - java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > -
> >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > - java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > - java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > -
> > java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1110)
> > -
> > java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:603)
> > - java.lang.Thread.run(Thread.java:679)
> >
> >
> >
> > Apparently I do see lots of threads like above in the thread dump. I'm
> > using latest build from the trunk (Apr 10th). Any insights into this
> issue
> > woudl be really helpful. Thanks a lot.
> >
>
Re: solr hangs
Posted by Pawel Rog <pa...@gmail.com>.
You wrote that you can see such error "OutOfMemoryError". I had such
problems when my caches were to big. It means that there is no more free
memory in JVM and probably full gc starts running. How big is your Java
heap? Maybe cache sizes in yout solr are to big according to your JVM
settings.
--
Regards,
Pawel
On Tue, Apr 10, 2012 at 9:51 PM, Peter Markey <su...@gmail.com> wrote:
> Hello,
>
> I have a solr cloud setup based on a blog (
> http://outerthought.org/blog/491-ot.html) and am able to bring up the
> instances and cores. But when I start indexing data (through csv update),
> the core throws a out of memory exception (null:java.lang.RuntimeException:
> java.lang.OutOfMemoryError: unable to create new native thread). The thread
> dump from new solr ui is below:
>
> cmdDistribExecutor-8-thread-777 (827)
>
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bd11b79
>
> - sun.misc.Unsafe.park(Native Method)
> - java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> -
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2043)
> -
>
> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
> -
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking
> (ConnPoolByRoute.java:403)
> -
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry
> (ConnPoolByRoute.java:300)
> -
>
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection
> (ThreadSafeClientConnManager.java:224)
> -
> org.apache.http.impl.client.DefaultRequestDirector.execute
> (DefaultRequestDirector.java:401)
> -
> org.apache.http.impl.client.AbstractHttpClient.execute
> (AbstractHttpClient.java:820)
> -
> org.apache.http.impl.client.AbstractHttpClient.execute
> (AbstractHttpClient.java:754)
> -
> org.apache.http.impl.client.AbstractHttpClient.execute
> (AbstractHttpClient.java:732)
> -
> org.apache.solr.client.solrj.impl.HttpSolrServer.request
> (HttpSolrServer.java:304)
> -
> org.apache.solr.client.solrj.impl.HttpSolrServer.request
> (HttpSolrServer.java:209)
> -
> org.apache.solr.update.SolrCmdDistributor$1.call
> (SolrCmdDistributor.java:320)
> -
> org.apache.solr.update.SolrCmdDistributor$1.call
> (SolrCmdDistributor.java:301)
> - java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> - java.util.concurrent.FutureTask.run(FutureTask.java:166)
> -
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> - java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> - java.util.concurrent.FutureTask.run(FutureTask.java:166)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1110)
> -
> java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:603)
> - java.lang.Thread.run(Thread.java:679)
>
>
>
> Apparently I do see lots of threads like above in the thread dump. I'm
> using latest build from the trunk (Apr 10th). Any insights into this issue
> woudl be really helpful. Thanks a lot.
>