You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "tomlewlit (JIRA)" <ji...@apache.org> on 2016/12/16 08:55:58 UTC
[jira] [Created] (SOLR-9871) Java-level deadlock on
SolrCore.getSearcher
tomlewlit created SOLR-9871:
-------------------------------
Summary: Java-level deadlock on SolrCore.getSearcher
Key: SOLR-9871
URL: https://issues.apache.org/jira/browse/SOLR-9871
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Affects Versions: 6.2.1
Reporter: tomlewlit
I have a SolrCloud setup with 2 nodes, 4 shards, 2 replicas, ~200mln documents. Sometimes one of the nodes hangs, jstack tells about java-level deadlock:
{noformat}
Found one Java-level deadlock:
=============================
"qtp1543727556-8467":
waiting to lock monitor 0x00007f5b6c9d8378 (object 0x000000050dc96730, a java.lang.Object),
which is held by "searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1 c:mail r:core_node8"
"searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1 c:mail r:core_node8":
waiting to lock monitor 0x00007f5aac41cd38 (object 0x000000050ded63e8, a org.apache.solr.update.SolrIndexWriter),
which is held by "commitScheduler-21-thread-1"
"commitScheduler-21-thread-1":
waiting to lock monitor 0x00007f5b6c9d8378 (object 0x000000050dc96730, a java.lang.Object),
which is held by "searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1 c:mail r:core_node8"
Java stack information for the threads listed above:
===================================================
"qtp1543727556-8467":
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1749)
- waiting to lock <0x000000050dc96730> (a java.lang.Object)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1552)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1487)
at org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:115)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:308)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"searcherExecutor-8-thread-1-processing-n:172.19.123.3:18984_solr x:mail_shard1_replica2 s:shard1 c:mail r:core_node8":
at org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:4974)
- waiting to lock <0x000000050ded63e8> (a org.apache.solr.update.SolrIndexWriter)
at org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:381)
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:251)
at org.apache.solr.search.SolrIndexSearcher.close(SolrIndexSearcher.java:461)
at org.apache.solr.core.SolrCore$2.close(SolrCore.java:1982)
at org.apache.solr.util.RefCounted.decref(RefCounted.java:56)
at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:2016)
- locked <0x000000050dc96730> (a java.lang.Object)
at org.apache.solr.core.SolrCore.lambda$getSearcher$5(SolrCore.java:1903)
at org.apache.solr.core.SolrCore$$Lambda$84/1360460105.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3/23599925.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"commitScheduler-21-thread-1":
at org.apache.solr.core.SolrCore.getIndexDir(SolrCore.java:302)
- waiting to lock <0x000000050dc96730> (a java.lang.Object)
at org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.isSnapshotted(SolrSnapshotMetaDataManager.java:232)
- locked <0x00000004d38deaa0> (a org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager)
at org.apache.solr.core.IndexDeletionPolicyWrapper$IndexCommitWrapper.delete(IndexDeletionPolicyWrapper.java:195)
at org.apache.solr.core.SolrDeletionPolicy.updateCommits(SolrDeletionPolicy.java:199)
- locked <0x0000000519863980> (a org.apache.solr.core.SolrDeletionPolicy)
at org.apache.solr.core.SolrDeletionPolicy.onCommit(SolrDeletionPolicy.java:101)
at org.apache.solr.core.IndexDeletionPolicyWrapper.onCommit(IndexDeletionPolicyWrapper.java:161)
at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:526)
at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3252)
- locked <0x000000050ded63e8> (a org.apache.solr.update.SolrIndexWriter)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3214)
- locked <0x000000050deed630> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3171)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:607)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org