You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Rogers <ch...@bodleian.ox.ac.uk> on 2016/08/30 20:34:51 UTC
SolrCloud: A previous ephemeral live node still exists
Hi all,
I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
But when I try and connect with a second node, something goes wrong.
I get the following error in the Solr log:
2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: /
2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs
2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already.
at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
It does start in the end, but it's not working properly, I just get errors in the Solr admin.
The only reference I can find to the the error:
"A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
https://issues.apache.org/jira/browse/SOLR-8777
Has anyone else experienced this issue? Any hints on resolving?
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>
RE: SolrCloud: A previous ephemeral live node still exists
Posted by Jon Hawkesworth <jo...@MEDQUIST.onmicrosoft.com>.
Hi Chris,
Not had this myself but is it possible there is still a 6.1 instance running on your solr nodes (which happened on my first attempt to update 6.1 to 6.2 yesterday)?
So check for running java processes.
Hope this helps,
Jon
-----Original Message-----
From: Chris Rogers [mailto:chris.rogers@bodleian.ox.ac.uk]
Sent: Wednesday, August 31, 2016 9:34 AM
To: solr-user@lucene.apache.org
Subject: RE: SolrCloud: A previous ephemeral live node still exists
Just pinging this again as I sent it late last night. Would be great if someone could help with this. It's got me totally stumped...
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services chris.rogers@bodleian.ox.ac.uk
________________________________________
From: Chris Rogers [chris.rogers@bodleian.ox.ac.uk]
Sent: 30 August 2016 21:34
To: solr-user@lucene.apache.org
Subject: SolrCloud: A previous ephemeral live node still exists
Hi all,
I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
But when I try and connect with a second node, something goes wrong.
I get the following error in the Solr log:
2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: /
2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs
2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already.
at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
It does start in the end, but it's not working properly, I just get errors in the Solr admin.
The only reference I can find to the the error:
"A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
https://issues.apache.org/jira/browse/SOLR-8777
Has anyone else experienced this issue? Any hints on resolving?
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>
Re: SolrCloud: A previous ephemeral live node still exists
Posted by Chris Rogers <ch...@bodleian.ox.ac.uk>.
Thanks Alan,
This turned out to be the problem. I was trying to connect to Zookeeper with the command:
bin/solr start -c -z 172.28.128.3:2181
with all of the nodes. So I think all the hostnames where defaulting to localhost. Specifying the hostname and port with the –h command fixed the issue.
I think I must have got this right the first time when testing with Solr 6.1.0, so my assumption that this was an issues with 6.2.0 seems to be a red herring.
Thanks again,
Chris
On 31/08/2016, 09:43, "Alan Woodward" <al...@flax.co.uk> wrote:
It looks as though all four nodes are trying to register with ZK using the same hostname and port number - possibly they're all connecting as 'localhost'?
Alan Woodward
www.flax.co.uk
On 31 Aug 2016, at 09:34, Chris Rogers wrote:
> Just pinging this again as I sent it late last night. Would be great if someone could help with this. It's got me totally stumped...
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk
>
> ________________________________________
> From: Chris Rogers [chris.rogers@bodleian.ox.ac.uk]
> Sent: 30 August 2016 21:34
> To: solr-user@lucene.apache.org
> Subject: SolrCloud: A previous ephemeral live node still exists
>
> Hi all,
>
> I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
>
> I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
>
> I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
>
> The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
>
> But when I try and connect with a second node, something goes wrong.
>
> I get the following error in the Solr log:
>
>
> 2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
>
> 2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
>
> 2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
>
> 2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
>
> 2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: /
>
> 2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>
> 2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>
> 2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
>
> 2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
>
> 2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
>
> 2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
>
> 2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
>
> 2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
>
> 2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
>
> 2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>
> 2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>
> 2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
>
> 2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
>
> 2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>
> 2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
>
> 2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>
> 2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>
> 2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs
>
> 2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already.
>
> at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
>
> at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
>
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
>
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
>
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
>
> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
>
> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
>
> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
>
> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
>
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
>
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
>
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>
> at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>
> at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
>
> at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>
> at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>
> at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>
> at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>
> at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>
> at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
>
> at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
>
> at org.eclipse.jetty.server.Server.start(Server.java:405)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
>
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
>
> at org.eclipse.jetty.start.Main.start(Main.java:457)
>
> at org.eclipse.jetty.start.Main.main(Main.java:75)
>
>
> 2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>
> 2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
>
> It does start in the end, but it's not working properly, I just get errors in the Solr admin.
>
> The only reference I can find to the the error:
>
> "A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
>
> ,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
>
> https://issues.apache.org/jira/browse/SOLR-8777
>
> Has anyone else experienced this issue? Any hints on resolving?
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>
Re: SolrCloud: A previous ephemeral live node still exists
Posted by Alan Woodward <al...@flax.co.uk>.
It looks as though all four nodes are trying to register with ZK using the same hostname and port number - possibly they're all connecting as 'localhost'?
Alan Woodward
www.flax.co.uk
On 31 Aug 2016, at 09:34, Chris Rogers wrote:
> Just pinging this again as I sent it late last night. Would be great if someone could help with this. It's got me totally stumped...
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk
>
> ________________________________________
> From: Chris Rogers [chris.rogers@bodleian.ox.ac.uk]
> Sent: 30 August 2016 21:34
> To: solr-user@lucene.apache.org
> Subject: SolrCloud: A previous ephemeral live node still exists
>
> Hi all,
>
> I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
>
> I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
>
> I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
>
> The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
>
> But when I try and connect with a second node, something goes wrong.
>
> I get the following error in the Solr log:
>
>
> 2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
>
> 2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
>
> 2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
>
> 2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
>
> 2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: /
>
> 2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>
> 2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>
> 2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
>
> 2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
>
> 2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
>
> 2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
>
> 2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
>
> 2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
>
> 2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
>
> 2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>
> 2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>
> 2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
>
> 2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
>
> 2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>
> 2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
>
> 2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>
> 2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>
> 2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs
>
> 2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already.
>
> at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
>
> at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
>
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
>
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
>
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
>
> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
>
> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
>
> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
>
> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
>
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
>
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
>
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>
> at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>
> at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
>
> at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>
> at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>
> at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>
> at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>
> at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>
> at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
>
> at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
>
> at org.eclipse.jetty.server.Server.start(Server.java:405)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
>
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
>
> at org.eclipse.jetty.start.Main.start(Main.java:457)
>
> at org.eclipse.jetty.start.Main.main(Main.java:75)
>
>
> 2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>
> 2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
>
> It does start in the end, but it's not working properly, I just get errors in the Solr admin.
>
> The only reference I can find to the the error:
>
> "A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
>
> ,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
>
> https://issues.apache.org/jira/browse/SOLR-8777
>
> Has anyone else experienced this issue? Any hints on resolving?
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>
RE: SolrCloud: A previous ephemeral live node still exists
Posted by Chris Rogers <ch...@bodleian.ox.ac.uk>.
Just pinging this again as I sent it late last night. Would be great if someone could help with this. It's got me totally stumped...
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rogers@bodleian.ox.ac.uk
________________________________________
From: Chris Rogers [chris.rogers@bodleian.ox.ac.uk]
Sent: 30 August 2016 21:34
To: solr-user@lucene.apache.org
Subject: SolrCloud: A previous ephemeral live node still exists
Hi all,
I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
But when I try and connect with a second node, something goes wrong.
I get the following error in the Solr log:
2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: /
2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs
2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already.
at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
It does start in the end, but it's not working properly, I just get errors in the Solr admin.
The only reference I can find to the the error:
"A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
https://issues.apache.org/jira/browse/SOLR-8777
Has anyone else experienced this issue? Any hints on resolving?
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rogers@bodleian.ox.ac.uk<ma...@bodleian.ox.ac.uk>