You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Darren Spehr (JIRA)" <ji...@apache.org> on 2015/01/18 22:14:34 UTC
[jira] [Created] (SOLR-6995) On a new cluster setup, new nodes
never move out of the 'down' state.
Darren Spehr created SOLR-6995:
----------------------------------
Summary: On a new cluster setup, new nodes never move out of the 'down' state.
Key: SOLR-6995
URL: https://issues.apache.org/jira/browse/SOLR-6995
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.10.3
Reporter: Darren Spehr
Priority: Blocker
This is related to a question I posted on [stackoverflow|http://stackoverflow.com/questions/28004832/solr-4-10-3-is-not-proceeding-to-leader-election-on-new-cluster-startup-hangs] a day ago.
When deploying a new cluster, new nodes never proceed to an 'active' state. I have used both a custom (yet simple) deployment, as well as using the example framework you get when downloading the Solr distribution. Further inspection of the logs, and comparing them to the output of a 4.10.1 start up, it looks like the nodes never move on to leader election.
For an apples-to-apples comparison I moved the solr/collection1 directory to solr/ttPoiMDS, and modified the core.properties appropriately. I also removed the 'conf' directory, assuming that it would pick this up from our ZooKeeper cluster.
Here is the start up:
{code}
java
-Dcollection.configName=ttPoiMDS
-Djava.util.logging.config.file=etc/logging.properties
-DzkHost=our_zk_host_1:<zk_port>
-Djetty.port=8983
-jar start.jar
{code}
Here is the output in the logs:
{panel}
0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
32 [main] INFO org.eclipse.jetty.deploy.providers.ScanningAppProvider – Deployment monitor /site/pkgs/solr/solr-4.10.3/example/contexts at interval 0
38 [main] INFO org.eclipse.jetty.deploy.DeploymentManager – Deployable added: /site/pkgs/solr/solr-4.10.3/example/contexts/solr-jetty-context.xml
114 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration – Extract jar:file:/site/pkgs/solr/solr-4.10.3/example/webapps/solr.war!/ to /site/pkgs/solr/solr-4.10.3/example/solr-webapp/webapp
1527 [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor – NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
1597 [main] INFO org.apache.solr.servlet.SolrDispatchFilter – SolrDispatchFilter.init()
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – JNDI not configured for solr (NoInitialContextEx)
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – solr home defaulted to 'solr/' (could not find system property or JNDI)
1617 [main] INFO org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for directory: 'solr/'
1748 [main] INFO org.apache.solr.core.ConfigSolr – Loading container configuration from /site/pkgs/solr/solr-4.10.3/example/solr/solr.xml
1884 [main] INFO org.apache.solr.core.CoresLocator – Config-defined core root directory: /site/pkgs/solr/solr-4.10.3/example/solr
1890 [main] INFO org.apache.solr.core.CoreContainer – New CoreContainer 378496804
1890 [main] INFO org.apache.solr.core.CoreContainer – Loading cores into CoreContainer [instanceDir=solr/]
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting socketTimeout to: 0
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting urlScheme to: null
1909 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting connTimeout to: 0
1911 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting maxConnectionsPerHost to: 20
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting corePoolSize to: 0
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting maximumPoolSize to: 2147483647
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting maxThreadIdleTime to: 5
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting sizeOfQueue to: -1
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory – Setting fairnessPolicy to: false
2054 [main] INFO org.apache.solr.update.UpdateShardHandler – Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
2056 [main] INFO org.apache.solr.logging.LogWatcher – SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2057 [main] INFO org.apache.solr.logging.LogWatcher – Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2058 [main] INFO org.apache.solr.core.CoreContainer – Host Name:
2059 [main] INFO org.apache.solr.core.ZkContainer – Zookeeper client=our_zk_host_1:2195
2140 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Waiting for client to connect to ZooKeeper
2220 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager – Watcher org.apache.solr.common.cloud.ConnectionManager@36dadde6 name:ZooKeeperConnection Watcher:our_zk_host_1:2195 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2220 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Client is connected to ZooKeeper
2293 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/queue
2447 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/collection-queue-work
2609 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/collection-map-running
2758 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/collection-map-completed
2919 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/collection-map-failure
3093 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /live_nodes
3173 [main] INFO org.apache.solr.cloud.ZkController – Register node as live in ZooKeeper:/live_nodes/our_solr_host:8983_solr
3201 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /live_nodes/our_solr_host:8983_solr
3371 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer_elect
3477 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer_elect/election
3577 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=null) closing
3659 [main] INFO org.apache.solr.cloud.ElectionContext – I am going to be the leader our_solr_host:8983_solr
3662 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer_elect/leader
3763 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=93155855038349319-our_solr_host:8983_solr-n_0000000000) starting
3887 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /overseer/queue-work
4409 [main] INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread – Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
4438 [OverseerCollectionProcessor-93155855038349319-our_solr_host:8983_solr-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor – Process current queue of collection creations
4456 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath: /clusterstate.json
4559 [main] INFO org.apache.solr.common.cloud.ZkStateReader – Updating cluster state from ZooKeeper...
4731 [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000] INFO org.apache.solr.cloud.Overseer – Starting to work on the main queue
4736 [main] INFO org.apache.solr.core.CoresLocator – Looking for core definitions underneath /site/pkgs/solr/solr-4.10.3/example/solr
4744 [main] INFO org.apache.solr.core.CoresLocator – Found core ttPoiMDS in /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/
4744 [main] INFO org.apache.solr.core.CoresLocator – Found 1 core definitions
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – publishing core=ttPoiMDS state=down collection=ttPoiMDS
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – numShards not found on descriptor - reading it from system property
4796 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – look for our core node name
4797 [zkCallback-2-thread-1] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
4897 [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000] INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
"operation":"state",
"shard":null,
"roles":null,
"state":"down",
"core":"ttPoiMDS",
"collection":"ttPoiMDS",
"node_name":"our_solr_host:8983_solr",
"base_url":"http://our_solr_host:8983/solr"}
4899 [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000] INFO org.apache.solr.cloud.Overseer – Assigning new node to shard shard=shard1
5052 [zkCallback-2-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader – A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
5800 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – waiting to find shard id in clusterstate for ttPoiMDS
5801 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – Check for collection zkNode:ttPoiMDS
5924 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController – Collection zkNode exists
5925 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader – Load collection config from:/collections/ttPoiMDS
5970 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader – path=/collections/ttPoiMDS configName=ttPoiMDS specified config exists in ZooKeeper
5970 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for directory: '/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/'
6054 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig – Adding specified lib dirs to ClassLoader
6055 [coreLoadExecutor-6-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../lib/mq (resolved as: /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/../../lib/mq).
6139 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig – Using Lucene MatchVersion: 4.10.3
6238 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.Config – Loaded SolrConfig: solrconfig.xml
6320 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema – Reading Solr Schema from /configs/ttPoiMDS/schema.xml
6347 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema – [ttPoiMDS] Schema name=poiMDS
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter – user.dir=/site/pkgs/solr/solr-4.10.3/example
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter – SolrDispatchFilter.init() done
6614 [main] INFO org.eclipse.jetty.server.AbstractConnector – Started SocketConnector@0.0.0.0:8983
{panel}
Startup then seems to hang. The Solr UI is available, but it shows that no cores are available. In the cloud view the configuration from ZooKeeper is present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org