You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Satyaprashant Bezwada <Sa...@nasdaq.com> on 2017/09/19 23:48:52 UTC
Solr replication
Need some inputs or help in resolving replication across solr nodes. We have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes configured. Enabled Solr replication in my Solrj client but the replication fails and is unable to create a collection.
The same code works in our different environment where in I have 1 zookeeper and 3 Solr nodes configured. Here is the exception I see on one of the nodes of Solr when I try to create a collection in the environment where it fails. I have compared the Solrconfig.xml on both the environments and didn’t see any difference.
2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$
Corp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
at java.lang.Thread.run(Thread.java:745)
2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /var/solr/data/Node122/solr.xml
2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=zk01:2181,zk02:2181,zk03:2181
2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer (id=null) closing
2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/sr01:8983_solr
2017-09-19 23:12:22.788 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2017-09-19 23:12:23.241 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data/Node122
2017-09-19 23:12:23.374 INFO (main) [ ] o.e.j.s.Server Started @2345ms
2017-09-19 23:12:47.125 INFO (qtp401424608-13) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=meetx&newCollection=true&name=CIUZLW_shard1_replica1&action=CREATE&numShards=1&collection=CIUZLW&shard=shard1&wt=javabin&version=2
2017-09-19 23:12:47.256 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
2017-09-19 23:12:48.148 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
2017-09-19 23:12:48.175 ERROR (qtp401424608-13) [c:CIUZLW s:shard1 x:CIUZLW_shard1_replica1] o.a.s.c.Config Exception during parsing file: solrconfig.xml:org.xml.sax.SAXParseException; systemId: solrres:/solrconfig.xml; lineNumber: 135; columnNumber: 13; Open quote is expected for attribute "name" associated with an element type "str".
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanAttributeValue(XMLScanner.java:830)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanAttribute(XMLNSDocumentScannerImpl.java:440)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:256)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at org.apache.solr.core.Config.<init>(Config.java:143)
at org.apache.solr.core.Config.<init>(Config.java:90)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:96)
at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:915)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:854)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:370)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
***********************************************************
Re: Solr replication
Posted by Satyaprashant Bezwada <Sa...@nasdaq.com>.
Thanks Eric, fixed the issue. The IT team corrected the solrconfig.xml but forgot to execute the zkcli.sh script on solr node. After I executed the script its working now.
On 9/20/17, 10:20 AM, "Erick Erickson" <er...@gmail.com> wrote:
WARNING - External email; exercise caution.
Your solrconfig.xml file is mal-formed. The smoking gun is:
Exception during parsing file: solrconfig.xml
Best,
Erick
On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
<Sa...@nasdaq.com> wrote:
> Need some inputs or help in resolving replication across solr nodes. We have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes configured. Enabled Solr replication in my Solrj client but the replication fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 zookeeper and 3 Solr nodes configured. Here is the exception I see on one of the nodes of Solr when I try to create a collection in the environment where it fails. I have compared the Solrconfig.xml on both the environments and didn’t see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
> 2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /var/solr/data/Node122/solr.xml
> 2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
> 2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=zk01:2181,zk02:2181,zk03:2181
> 2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
> 2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer (id=null) closing
> 2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/sr01:8983_solr
> 2017-09-19 23:12:22.788 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> 2017-09-19 23:12:23.241 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data/Node122
> 2017-09-19 23:12:23.374 INFO (main) [ ] o.e.j.s.Server Started @2345ms
> 2017-09-19 23:12:47.125 INFO (qtp401424608-13) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=meetx&newCollection=true&name=CIUZLW_shard1_replica1&action=CREATE&numShards=1&collection=CIUZLW&shard=shard1&wt=javabin&version=2
> 2017-09-19 23:12:47.256 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.148 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.175 ERROR (qtp401424608-13) [c:CIUZLW s:shard1 x:CIUZLW_shard1_replica1] o.a.s.c.Config Exception during parsing file: solrconfig.xml:org.xml.sax.SAXParseException; systemId: solrres:/solrconfig.xml; lineNumber: 135; columnNumber: 13; Open quote is expected for attribute "name" associated with an element type "str".
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanAttributeValue(XMLScanner.java:830)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanAttribute(XMLNSDocumentScannerImpl.java:440)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:256)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at org.apache.solr.core.Config.<init>(Config.java:143)
> at org.apache.solr.core.Config.<init>(Config.java:90)
> at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
> at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
> at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:96)
> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:915)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:854)
> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:370)
> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:745)
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
> ***********************************************************
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
***********************************************************
Re: Solr replication
Posted by Erick Erickson <er...@gmail.com>.
Your solrconfig.xml file is mal-formed. The smoking gun is:
Exception during parsing file: solrconfig.xml
Best,
Erick
On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
<Sa...@nasdaq.com> wrote:
> Need some inputs or help in resolving replication across solr nodes. We have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes configured. Enabled Solr replication in my Solrj client but the replication fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 zookeeper and 3 Solr nodes configured. Here is the exception I see on one of the nodes of Solr when I try to create a collection in the environment where it fails. I have compared the Solrconfig.xml on both the environments and didn’t see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezwada@boardvantage.net@sr1501:1 ~]$ sudo tail -f /var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
> 2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /var/solr/data/Node122/solr.xml
> 2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
> 2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=zk01:2181,zk02:2181,zk03:2181
> 2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
> 2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer (id=null) closing
> 2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/sr01:8983_solr
> 2017-09-19 23:12:22.788 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> 2017-09-19 23:12:23.241 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data/Node122
> 2017-09-19 23:12:23.374 INFO (main) [ ] o.e.j.s.Server Started @2345ms
> 2017-09-19 23:12:47.125 INFO (qtp401424608-13) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=meetx&newCollection=true&name=CIUZLW_shard1_replica1&action=CREATE&numShards=1&collection=CIUZLW&shard=shard1&wt=javabin&version=2
> 2017-09-19 23:12:47.256 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.148 INFO (zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json] for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.175 ERROR (qtp401424608-13) [c:CIUZLW s:shard1 x:CIUZLW_shard1_replica1] o.a.s.c.Config Exception during parsing file: solrconfig.xml:org.xml.sax.SAXParseException; systemId: solrres:/solrconfig.xml; lineNumber: 135; columnNumber: 13; Open quote is expected for attribute "name" associated with an element type "str".
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
> at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
> at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanAttributeValue(XMLScanner.java:830)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanAttribute(XMLNSDocumentScannerImpl.java:440)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:256)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at org.apache.solr.core.Config.<init>(Config.java:143)
> at org.apache.solr.core.Config.<init>(Config.java:90)
> at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
> at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
> at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:96)
> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:915)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:854)
> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:370)
> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:745)
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
> ***********************************************************