You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zafar Khurasani <Za...@Rentacenter.com> on 2018/08/27 14:52:21 UTC
Issue with adding an extra Solr Slave
Hi,
I'm running Solr 5.3 in one of our applications. Currently, we have one Solr Master and one Solr slave running on AWS EC2 instances. I'm trying to add an additional Solr slave. I'm using an Elastic LoadBalancer (ELB) in front of my Slaves. I see the following error in the logs after adding the second slave,
java version "1.8.0_121"
Solr version: 5.3.0 1696229
org.apache.solr.common.SolrException: Core with core name [xxx-xxx-xxxx] does not exist.
at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:240)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:194)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:675)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Also, when I hit the Solr Admin UI, I'm able to see my core infrequently. I have to refresh the page multiple times to be able to see it. What's the right way to add a slave to my existing setup?
FYI - the Solr Replication section in solrconfig.xml is exactly the same for both the Slaves.
Thanks,
Zafar Khurasani
Re: Issue with adding an extra Solr Slave
Posted by Emir Arnautović <em...@sematext.com>.
Hi Zafar,
Slaves are separate nodes and accessing admin console through ELB does not make much sense since different requests will go to different nodes and that’s why you sometimes see cores and other time it is empty. Since it is empty, it seems that you did not define core(s) on this new slave. Replication handler is defined on core level so I am not sure what you mean that solrconfig.xml are the same on both servers.
What you need to do is create new core on new slave. Make sure replication handler is properly configured and that master is reachable (try pinging replication handler of master from slave). Issue fetch index command for new slave (http://slave_host:port/solr/core_name/replication?command=fetchindex). And when checking in admin console, use slave’s IP, not ELB.
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 28 Aug 2018, at 21:03, Zafar Khurasani <Za...@Rentacenter.com> wrote:
>
> Hi Emir,
>
> I access the admin console through the ELB. I do NOT see any replication errors in the new Slave's logs. I also double checked to make sure the connectivity between the master and slaves exist. The only error I see in the new Slave log is what I shared originally.
>
> Thanks,
> Zafar.
>
>
>
> -----Original Message-----
> From: Emir Arnautović [mailto:emir.arnautovic@sematext.com]
> Sent: Tuesday, August 28, 2018 2:55 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Issue with adding an extra Solr Slave
>
> Hi Zafar,
> How do you access admin console? Through ELB or you see this behaviour when accessing admin console of a new slave? Do you see any replication related errors in new slave’s logs? Did you check connectivity of a new slave and master nodes?
>
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 27 Aug 2018, at 16:52, Zafar Khurasani <Za...@Rentacenter.com> wrote:
>>
>> Hi,
>>
>> I'm running Solr 5.3 in one of our applications. Currently, we have
>> one Solr Master and one Solr slave running on AWS EC2 instances. I'm
>> trying to add an additional Solr slave. I'm using an Elastic
>> LoadBalancer (ELB) in front of my Slaves. I see the following error in
>> the logs after adding the second slave,
>>
>>
>> java version "1.8.0_121"
>>
>> Solr version: 5.3.0 1696229
>>
>>
>> org.apache.solr.common.SolrException: Core with core name [xxx-xxx-xxxx] does not exist.
>> at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
>> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:240)
>> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:194)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:675)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Also, when I hit the Solr Admin UI, I'm able to see my core infrequently. I have to refresh the page multiple times to be able to see it. What's the right way to add a slave to my existing setup?
>>
>> FYI - the Solr Replication section in solrconfig.xml is exactly the same for both the Slaves.
>>
>> Thanks,
>> Zafar Khurasani
>>
>
RE: Issue with adding an extra Solr Slave
Posted by Zafar Khurasani <Za...@Rentacenter.com>.
Hi Emir,
I access the admin console through the ELB. I do NOT see any replication errors in the new Slave's logs. I also double checked to make sure the connectivity between the master and slaves exist. The only error I see in the new Slave log is what I shared originally.
Thanks,
Zafar.
-----Original Message-----
From: Emir Arnautović [mailto:emir.arnautovic@sematext.com]
Sent: Tuesday, August 28, 2018 2:55 AM
To: solr-user@lucene.apache.org
Subject: Re: Issue with adding an extra Solr Slave
Hi Zafar,
How do you access admin console? Through ELB or you see this behaviour when accessing admin console of a new slave? Do you see any replication related errors in new slave’s logs? Did you check connectivity of a new slave and master nodes?
Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Aug 2018, at 16:52, Zafar Khurasani <Za...@Rentacenter.com> wrote:
>
> Hi,
>
> I'm running Solr 5.3 in one of our applications. Currently, we have
> one Solr Master and one Solr slave running on AWS EC2 instances. I'm
> trying to add an additional Solr slave. I'm using an Elastic
> LoadBalancer (ELB) in front of my Slaves. I see the following error in
> the logs after adding the second slave,
>
>
> java version "1.8.0_121"
>
> Solr version: 5.3.0 1696229
>
>
> org.apache.solr.common.SolrException: Core with core name [xxx-xxx-xxxx] does not exist.
> at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:240)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:194)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:675)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Also, when I hit the Solr Admin UI, I'm able to see my core infrequently. I have to refresh the page multiple times to be able to see it. What's the right way to add a slave to my existing setup?
>
> FYI - the Solr Replication section in solrconfig.xml is exactly the same for both the Slaves.
>
> Thanks,
> Zafar Khurasani
>
Re: Issue with adding an extra Solr Slave
Posted by Emir Arnautović <em...@sematext.com>.
Hi Zafar,
How do you access admin console? Through ELB or you see this behaviour when accessing admin console of a new slave? Do you see any replication related errors in new slave’s logs? Did you check connectivity of a new slave and master nodes?
Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Aug 2018, at 16:52, Zafar Khurasani <Za...@Rentacenter.com> wrote:
>
> Hi,
>
> I'm running Solr 5.3 in one of our applications. Currently, we have one Solr Master and one Solr slave running on AWS EC2 instances. I'm trying to add an additional Solr slave. I'm using an Elastic LoadBalancer (ELB) in front of my Slaves. I see the following error in the logs after adding the second slave,
>
>
> java version "1.8.0_121"
>
> Solr version: 5.3.0 1696229
>
>
> org.apache.solr.common.SolrException: Core with core name [xxx-xxx-xxxx] does not exist.
> at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:240)
> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:194)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:675)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Also, when I hit the Solr Admin UI, I'm able to see my core infrequently. I have to refresh the page multiple times to be able to see it. What's the right way to add a slave to my existing setup?
>
> FYI - the Solr Replication section in solrconfig.xml is exactly the same for both the Slaves.
>
> Thanks,
> Zafar Khurasani
>