You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Joseph Duchesne (JIRA)" <ji...@apache.org> on 2014/02/14 16:28:20 UTC
[jira] [Comment Edited] (SOLR-5724) Two node, one shard solr
instance intermittently going offline
[ https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901543#comment-13901543 ]
Joseph Duchesne edited comment on SOLR-5724 at 2/14/14 3:26 PM:
----------------------------------------------------------------
Here are the logs from replica 1:
{quote}
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error while trying to recover. core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: We are not the leader
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
And replica 2:
{quote}
ERROR - 2014-02-13 13:39:01.143; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 1
ERROR - 2014-02-13 13:39:01.158; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error while trying to recover. core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: We are not the leader
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ... retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
request: http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
ERROR - 2014-02-13 13:39:02.167; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I have saved the full log files, but they're over 4MB each.
> Two node, one shard solr instance intermittently going offline
> ---------------------------------------------------------------
>
> Key: SOLR-5724
> URL: https://issues.apache.org/jira/browse/SOLR-5724
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.6.1
> Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> Reporter: Joseph Duchesne
>
> One server is stuck in state "recovering" while the other is stuck in state "down". After waiting 45 minutes or so for the cluster to recover, the statuses were the same.
> Log messages on the "recovering" server: (Just the individual errors for brevity, I can provide full stack traces if that is helpful)
> {quote}
> We are not the leader
> ClusterState says we are the leader, but locally we don't think so
> cancelElection did not find election node to remove
> We are not the leader
> No registered leader was found, collection:listsC slice:shard1
> No registered leader was found, collection:listsC slice:shard1
> {quote}
> On the "down" server at the same timeframe:
> {quote}
> org.apache.solr.common.SolrException; forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
> org.apache.solr.update.StreamingSolrServers$1; error
> Error while trying to recover. core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: We are not the leader
> Recovery failed - trying again... (0) core=listsC
> Stopping recovery for zkNodeName=core_node2core=listsC
> org.apache.solr.update.StreamingSolrServers$1; error
> org.apache.solr.common.SolrException: Service Unavailable
> {quote}
> I am not sure what is causing this, however it has happened a 3 times in the past week. If there are any additional logs I can provide, or if there is anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org